diff options
Diffstat (limited to 'doc/architecture/blueprints/cells')
62 files changed, 2260 insertions, 1680 deletions
diff --git a/doc/architecture/blueprints/cells/cells-feature-admin-area.md b/doc/architecture/blueprints/cells/cells-feature-admin-area.md index a9cd170b2a7..3f23e56c3af 100644 --- a/doc/architecture/blueprints/cells/cells-feature-admin-area.md +++ b/doc/architecture/blueprints/cells/cells-feature-admin-area.md @@ -1,81 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Admin Area' +redirect_to: 'impacted_features/admin-area.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Admin Area - -In our Cells architecture proposal we plan to share all admin related tables in GitLab. -This allows for simpler management of all Cells in one interface and reduces the risk of settings diverging in different Cells. -This introduces challenges with Admin Area pages that allow you to manage data that will be spread across all Cells. - -## 1. Definition - -There are consequences for Admin Area pages that contain data that span "the whole instance" as the Admin Area pages may be served by any Cell or possibly just one Cell. -There are already many parts of the Admin Area that will have data that span many Cells. -For example lists of all Groups, Projects, Topics, Jobs, Analytics, Applications and more. -There are also administrative monitoring capabilities in the Admin Area that will span many Cells such as the "Background Jobs" and "Background Migrations" pages. - -## 2. Data flow - -## 3. Proposal - -We will need to decide how to handle these exceptions with a few possible -options: - -1. Move all these pages out into a dedicated per-Cell admin section. Probably - the URL will need to be routable to a single Cell like `/cells/<cell_id>/admin`, - then we can display these data per Cell. These pages will be distinct from - other Admin Area pages which control settings that are shared across all Cells. We - will also need to consider how this impacts self-managed customers and - whether, or not, this should be visible for single-Cell instances of GitLab. -1. Build some aggregation interfaces for this data so that it can be fetched - from all Cells and presented in a single UI. This may be beneficial to an - administrator that needs to see and filter all data at a glance, especially - when they don't know which Cell the data is on. The downside, however, is - that building this kind of aggregation is very tricky when all Cells are - designed to be totally independent, and it does also enforce stricter - requirements on compatibility between Cells. - -The following overview describes at what level each feature contained in the current Admin Area will be managed: - -| Feature | Cluster | Cell | Organization | -| --- | --- | --- | --- | -| Abuse reports | | | | -| Analytics | | | | -| Applications | | | | -| Deploy keys | | | | -| Labels | | | | -| Messages | ✓ | | | -| Monitoring | | ✓ | | -| Subscription | | | | -| System hooks | | | | -| Overview | | | | -| Settings - General | ✓ | | | -| Settings - Integrations | ✓ | | | -| Settings - Repository | ✓ | | | -| Settings - CI/CD (1) | ✓ | ✓ | | -| Settings - Reporting | ✓ | | | -| Settings - Metrics | ✓ | | | -| Settings - Service usage data | | ✓ | | -| Settings - Network | ✓ | | | -| Settings - Appearance | ✓ | | | -| Settings - Preferences | ✓ | | | - -(1) Depending on the specific setting, some will be managed at the cluster-level, and some at the Cell-level. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/admin-area.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-agent-for-kubernetes.md b/doc/architecture/blueprints/cells/cells-feature-agent-for-kubernetes.md index 37347cf836d..050b3a922b1 100644 --- a/doc/architecture/blueprints/cells/cells-feature-agent-for-kubernetes.md +++ b/doc/architecture/blueprints/cells/cells-feature-agent-for-kubernetes.md @@ -1,30 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Agent for Kubernetes' +redirect_to: 'impacted_features/agent-for-kubernetes.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Agent for Kubernetes - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/agent-for-kubernetes.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-backups.md b/doc/architecture/blueprints/cells/cells-feature-backups.md index 3d20d6e2caa..a0c38171ce6 100644 --- a/doc/architecture/blueprints/cells/cells-feature-backups.md +++ b/doc/architecture/blueprints/cells/cells-feature-backups.md @@ -1,53 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Backups' +redirect_to: 'impacted_features/backups.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Backups - -Each Cell will take its own backups, and consequently have its own isolated backup/restore procedure. - -## 1. Definition - -GitLab backup takes a backup of the PostgreSQL database used by the application, and also Git repository data. - -## 2. Data flow - -Each Cell has a number of application databases to back up (for example, `main`, and `ci`). -Additionally, there may be cluster-wide metadata tables (for example, `users` table) which is directly accessible via PostgreSQL. - -## 3. Proposal - -### 3.1. Cluster-wide metadata - -It is currently unknown how cluster-wide metadata tables will be accessible. -We may choose to have cluster-wide metadata tables backed up separately, or have each Cell back up its copy of cluster-wide metadata tables. - -### 3.2 Consistency - -#### 3.2.1 Take backups independently - -As each Cell will communicate with each other via API, and there will be no joins to the `users` table, it should be acceptable for each Cell to take a backup independently of each other. - -#### 3.2.2 Enforce snapshots - -We can require that each Cell take a snapshot for the PostgreSQL databases at around the same time to allow for a consistent enough backup. - -## 4. Evaluation - -As the number of Cells increases, it will likely not be feasible to take a snapshot at the same time for all Cells. -Hence taking backups independently is the better option. - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/backups.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-ci-runners.md b/doc/architecture/blueprints/cells/cells-feature-ci-runners.md index 4e7cea5bfd5..a14f2a47237 100644 --- a/doc/architecture/blueprints/cells/cells-feature-ci-runners.md +++ b/doc/architecture/blueprints/cells/cells-feature-ci-runners.md @@ -1,144 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: CI Runners' +redirect_to: 'impacted_features/ci-runners.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: CI Runners - -GitLab executes CI jobs via [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/), very often managed by customers in their infrastructure. -All CI jobs created as part of the CI pipeline are run in the context of a Project. -This poses a challenge how to manage GitLab Runners. - -## 1. Definition - -There are 3 different types of runners: - -- Instance-wide: Runners that are registered globally with specific tags (selection criteria) -- Group runners: Runners that execute jobs from a given top-level Group or Projects in that Group -- Project runners: Runners that execute jobs from one Projects or many Projects: some runners might - have Projects assigned from Projects in different top-level Groups. - -This, alongside with the existing data structure where `ci_runners` is a table describing all types of runners, poses a challenge as to how the `ci_runners` should be managed in a Cells environment. - -## 2. Data flow - -GitLab runners use a set of globally scoped endpoints to: - -- Register a new runner via registration token `https://gitlab.com/api/v4/runners` - ([subject for removal](../runner_tokens/index.md)) (`registration token`) -- Create a new runner in the context of a user `https://gitlab.com/api/v4/user/runners` (`runner token`) -- Request jobs via an authenticated `https://gitlab.com/api/v4/jobs/request` endpoint (`runner token`) -- Upload job status via `https://gitlab.com/api/v4/jobs/:job_id` (`build token`) -- Upload trace via `https://gitlab.com/api/v4/jobs/:job_id/trace` (`build token`) -- Download and upload artifacts via `https://gitlab.com/api/v4/jobs/:job_id/artifacts` (`build token`) - -Currently three types of authentication tokens are used: - -- Runner registration token ([subject for removal](../runner_tokens/index.md)) -- Runner token representing a registered runner in a system with specific configuration (`tags`, `locked`, etc.) -- Build token representing an ephemeral token giving limited access to updating a specific job, uploading artifacts, downloading dependent artifacts, downloading and uploading container registry images - -Each of those endpoints receive an authentication token via header (`JOB-TOKEN` for `/trace`) or body parameter (`token` all other endpoints). - -Since the CI pipeline would be created in the context of a specific Cell, it would be required that pick of a build would have to be processed by that particular Cell. -This requires that build picking depending on a solution would have to be either: - -- Routed to the correct Cell for the first time -- Be two-phased: Request build from global pool, claim build on a specific Cell using a Cell specific URL - -## 3. Proposal - -### 3.1. Authentication tokens - -Even though the paths for CI runners are not routable, they can be made routable with these two possible solutions: - -- The `https://gitlab.com/api/v4/jobs/request` uses a long polling mechanism with - a ticketing mechanism (based on `X-GitLab-Last-Update` header). When the runner first - starts, it sends a request to GitLab to which GitLab responds with either a build to pick - by runner. This value is completely controlled by GitLab. This allows GitLab - to use JWT or any other means to encode a `cell` identifier that could be easily - decodable by Router. -- The majority of communication (in terms of volume) is using `build token`, making it - the easiest target to change since GitLab is the sole owner of the token that the runner later - uses for a specific job. There were prior discussions about not storing the `build token` - but rather using a `JWT` token with defined scopes. Such a token could encode the `cell` - to which the Router could route all requests. - -### 3.2. Request body - -- The most used endpoints pass the authentication token in the request body. It might be desired - to use HTTP headers as an easier way to access this information by Router without - a need to proxy requests. - -### 3.3. Instance-wide are Cell-local - -We can pick a design where all runners are always registered and local to a given Cell: - -- Each Cell has its own set of instance-wide runners that are updated at its own pace -- The Project runners can only be linked to Projects from the same Organization, creating strong isolation. -- In this model the `ci_runners` table is local to the Cell. -- In this model we would require the above endpoints to be scoped to a Cell in some way, or be made routable. It might be via prefixing them, adding additional Cell parameters, or providing much more robust ways to decode runner tokens and match it to a Cell. -- If a routable token is used, we could move away from cryptographic random stored in database to rather prefer to use JWT tokens. -- The Admin Area showing registered runners would have to be scoped to a Cell. - -This model might be desired because it provides strong isolation guarantees. -This model does significantly increase maintenance overhead because each Cell is managed separately. -This model may require adjustments to the runner tags feature so that Projects have a consistent runner experience across Cells. - -### 3.4. Instance-wide are cluster-wide - -Contrary to the proposal where all runners are Cell-local, we can consider that runners -are global, or just instance-wide runners are global. - -However, this requires significant overhaul of the system and we would have to change the following aspects: - -- The `ci_runners` table would likely have to be decomposed into `ci_instance_runners`, ... -- All interfaces would have to be adopted to use the correct table. -- Build queuing would have to be reworked to be two-phased where each Cell would know of all pending and running builds, but the actual claim of a build would happen against a Cell containing data. -- It is likely that `ci_pending_builds` and `ci_running_builds` would have to be made `cluster-wide` tables, increasing the likelihood of creating hotspots in a system related to CI queueing. - -This model is complex to implement from an engineering perspective. -Some data are shared between Cells. -It creates hotspots/scalability issues in a system that might impact the experience of Organizations on other Cells, for instance during abuse. - -### 3.5. GitLab CI Daemon - -Another potential solution to explore is to have a dedicated service responsible for builds queueing, owning its database and working in a model of either sharded or Cell-ed service. -There were prior discussions about [CI/CD Daemon](https://gitlab.com/gitlab-org/gitlab/-/issues/19435). - -If the service is sharded: - -- Depending on the model, if runners are cluster-wide or Cell-local, this service would have to fetch data from all Cells. -- If the sharded service is used we could adapt a model of sharing a database containing `ci_pending_builds/ci_running_builds` with the service. -- If the sharded service is used we could consider a push model where each Cell pushes to CI/CD Daemon builds that should be picked by runner. -- The sharded service would be aware which Cell is responsible for processing the given build and could route processing requests to the designated Cell. - -If the service is Cell-ed: - -- All expectations of routable endpoints are still valid. - -In general usage of CI Daemon does not help significantly with the stated problem. -However, this offers a few upsides related to more efficient processing and decoupling model: push model and it opens a way to offer stateful communication with GitLab runners (ex. gRPC or Websockets). - -## 4. Evaluation - -Considering all options it appears that the most promising solution is to: - -- Use [Instance-wide are Cell-local](#33-instance-wide-are-cell-local) -- Refine endpoints to have routable identities (either via specific paths, or better tokens) - -Another potential upside is to get rid of `ci_builds.token` and rather use a `JWT token` that can much better and easier encode a wider set of scopes allowed by CI runner. - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/ci-runners.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-container-registry.md b/doc/architecture/blueprints/cells/cells-feature-container-registry.md index 25af65a8700..d9ff6da7f62 100644 --- a/doc/architecture/blueprints/cells/cells-feature-container-registry.md +++ b/doc/architecture/blueprints/cells/cells-feature-container-registry.md @@ -1,114 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Container Registry' +redirect_to: 'impacted_features/container-registry.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Container Registry - -GitLab [Container Registry](../../../user/packages/container_registry/index.md) is a feature allowing to store Docker container images in GitLab. - -## 1. Definition - -GitLab Container Registry is a complex service requiring usage of PostgreSQL, Redis and Object Storage dependencies. -Right now there's undergoing work to introduce [Container Registry Metadata](../container_registry_metadata_database/index.md) to optimize data storage and image retention policies of Container Registry. - -GitLab Container Registry is serving as a container for stored data, but on its own does not authenticate `docker login`. -The `docker login` is executed with user credentials (can be `personal access token`) or CI build credentials (ephemeral `ci_builds.token`). - -Container Registry uses data deduplication. -It means that the same blob (image layer) that is shared between many Projects is stored only once. -Each layer is hashed by `sha256`. - -The `docker login` does request a JWT time-limited authentication token that is signed by GitLab, but validated by Container Registry service. -The JWT token does store all authorized scopes (`container repository images`) and operation types (`push` or `pull`). -A single JWT authentication token can have many authorized scopes. -This allows Container Registry and client to mount existing blobs from other scopes. -GitLab responds only with authorized scopes. -Then it is up to GitLab Container Registry to validate if the given operation can be performed. - -The GitLab.com pages are always scoped to a Project. -Each Project can have many container registry images attached. - -Currently, on GitLab.com the actual registry service is served via `https://registry.gitlab.com`. - -The main identifiable problems are: - -- The authentication request (`https://gitlab.com/jwt/auth`) that is processed by GitLab.com. -- The `https://registry.gitlab.com` that is run by an external service and uses its own data store. -- Data deduplication. The Cells architecture with registry run in a Cell would reduce efficiency of data storage. - -## 2. Data flow - -### 2.1. Authorization request that is send by `docker login` - -```shell -curl \ - --user "username:password" \ - "https://gitlab/jwt/auth?client_id=docker&offline_token=true&service=container_registry&scope=repository:gitlab-org/gitlab-build-images:push,pull" -``` - -Result is encoded and signed JWT token. Second base64 encoded string (split by `.`) contains JSON with authorized scopes. - -```json -{"auth_type":"none","access":[{"type":"repository","name":"gitlab-org/gitlab-build-images","actions":["pull"]}],"jti":"61ca2459-091c-4496-a3cf-01bac51d4dc8","aud":"container_registry","iss":"omnibus-gitlab-issuer","iat":1669309469,"nbf":166} -``` - -### 2.2. Docker client fetching tags - -```shell -curl \ - -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ - -H "Authorization: Bearer token" \ - https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/tags/list - -curl \ - -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ - -H "Authorization: Bearer token" \ - https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/manifests/danger-ruby-2.6.6 -``` - -### 2.3. Docker client fetching blobs and manifests - -```shell -curl \ - -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ - -H "Authorization: Bearer token" \ - https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/blobs/sha256:a3f2e1afa377d20897e08a85cae089393daa0ec019feab3851d592248674b416 -``` - -## 3. Proposal - -### 3.1. Shard Container Registry separately to Cells architecture - -Due to its extensive and in general highly scalable horizontal architecture it should be evaluated if the GitLab Container Registry should be run not in Cell, but in a Cluster and be scaled independently. -This might be easier, but would definitely not offer the same amount of data isolation. - -### 3.2. Run Container Registry within a Cell - -It appears that except `/jwt/auth` which would likely have to be processed by Router (to decode `scope`) the Container Registry could be run as a local service of a Cell. -The actual data at least in case of GitLab.com is not forwarded via registry, but rather served directly from Object Storage / CDN. - -Its design encodes container repository image in a URL that is easily routable. -It appears that we could re-use the same stateless Router service in front of Container Registry to serve manifests and blobs redirect. - -The only downside is increased complexity of managing standalone registry for each Cell, but this might be desired approach. - -## 4. Evaluation - -There do not seem to be any theoretical problems with running GitLab Container Registry in a Cell. -It seems that the service can be easily made routable to work well. -The practical complexities are around managing a complex service from an infrastructure side. - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/container-registry.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-contributions-forks.md b/doc/architecture/blueprints/cells/cells-feature-contributions-forks.md index 8e144386908..a87e4ba3391 100644 --- a/doc/architecture/blueprints/cells/cells-feature-contributions-forks.md +++ b/doc/architecture/blueprints/cells/cells-feature-contributions-forks.md @@ -1,106 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Contributions: Forks' +redirect_to: 'impacted_features/contributions-forks.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Contributions: Forks - -The [Forking workflow](../../../user/project/repository/forking_workflow.md) allows users to copy existing Project sources into their own namespace of choice (Personal or Group). - -## 1. Definition - -The [Forking workflow](../../../user/project/repository/forking_workflow.md) is a common workflow with various usage patterns: - -- It allows users to contribute back to upstream Project. -- It persists repositories into their Personal Namespace. -- Users can copy to make changes and release as modified Project. - -Forks allow users not having write access to a parent Project to make changes. -The forking workflow is especially important for the open source community to contribute back to public Projects. -However, it is equally important in some companies that prefer a strong split of responsibilities and tighter access control. -The access to a Project is restricted to a designated list of developers. - -Forks enable: - -- Tighter control of who can modify the upstream Project. -- Split of responsibilities: Parent Project might use CI configuration connecting to production systems. -- To run CI pipelines in the context of a fork in a much more restrictive environment. -- To consider all forks to be unvetted which reduces risks of leaking secrets, or any other information tied to the Project. - -The forking model is problematic in a Cells architecture for the following reasons: - -- Forks are clones of existing repositories. Forks could be created across different Organizations, Cells and Gitaly shards. -- Users can create merge requests and contribute back to an upstream Project. This upstream Project might in a different Organization and Cell. -- The merge request CI pipeline is executed in the context of the source Project, but presented in the context of the target Project. - -## 2. Data flow - -## 3. Proposals - -### 3.1. Intra-Cluster forks - -This proposal implements forks as intra-Cluster forks where communication is done via API between all trusted Cells of a cluster: - -- Forks are created always in the context of a user's choice of Group. -- Forks are isolated from the Organization. -- Organization or Group owner could disable forking across Organizations, or forking in general. -- A merge request is created in the context of the target Project, referencing the external Project on another Cell. -- To target Project the merge reference is transferred that is used for presenting information in context of the target Project. -- CI pipeline is fetched in the context of the source Project as it is today, the result is fetched into the merge request of the target Project. -- The Cell holding the target Project internally uses GraphQL to fetch the status of the source Project and includes in context of the information for merge request. - -Upsides: - -- All existing forks continue to work as they are, as they are treated as intra-Cluster forks. - -Downsides: - -- The purpose of Organizations is to provide strong isolation between Organizations. Allowing to fork across does break security boundaries. -- However, this is no different to the ability of users today to clone a repository to a local computer and push it to any repository of choice. -- Access control of source Project can be lower than those of target Project. Today, the system requires that in order to contribute back, the access level needs to be the same for fork and upstream. - -### 3.2. Forks are created in a Personal Namespace of the current Organization - -Instead of creating Projects across Organizations, forks are created in a user's Personal Namespace tied to the Organization. Example: - -- Each user that is part of an Organization receives their Personal Namespace. For example for `GitLab Inc.` it could be `gitlab.com/organization/gitlab-inc/@ayufan`. -- The user has to fork into their own Personal Namespace of the Organization. -- The user has as many Personal Namespaces as Organizations they belongs to. -- The Personal Namespace behaves similar to the currently offered Personal Namespace. -- The user can manage and create Projects within a Personal Namespace. -- The Organization can prevent or disable usage of Personal Namespaces, disallowing forks. -- All current forks are migrated into the Personal Namespace of user in an Organization. -- All forks are part of the Organization. -- Forks are not federated features. -- The Personal Namespace and forked Project do not share configuration with the parent Project. - -### 3.3. Forks are created as internal Projects under current Projects - -Instead of creating Projects across Organizations, forks are attachments to existing Projects. -Each user forking a Project receives their unique Project. Example: - -- For Project: `gitlab.com/gitlab-org/gitlab`, forks would be created in `gitlab.com/gitlab-org/gitlab/@kamil-gitlab`. -- Forks are created in the context of the current Organization, they do not cross Organization boundaries and are managed by the Organization. -- Tied to the user (or any other user-provided name of the fork). -- Forks are not federated features. - -Downsides: - -- Does not answer how to handle and migrate all existing forks. -- Might share current Group/Project settings, which could be breaking some security boundaries. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/contributions-forks.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-data-migration.md b/doc/architecture/blueprints/cells/cells-feature-data-migration.md index 9ff661ddf68..5638bb29dc5 100644 --- a/doc/architecture/blueprints/cells/cells-feature-data-migration.md +++ b/doc/architecture/blueprints/cells/cells-feature-data-migration.md @@ -1,99 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Data migration' +redirect_to: 'impacted_features/data-migration.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Data migration - -It is essential for a Cells architecture to provide a way to migrate data out of big Cells into smaller ones. -This document describes various approaches to provide this type of split. - -We also need to handle cases where data is already violating the expected isolation constraints of Cells, for example references cannot span multiple Organizations. -We know that existing features like linked issues allowed users to link issues across any Projects regardless of their hierarchy. -There are many similar features. -All of this data will need to be migrated in some way before it can be split across different Cells. -This may mean some data needs to be deleted, or the feature needs to be changed and modelled slightly differently before we can properly split or migrate Organizations between Cells. - -Having schema deviations across different Cells, which is a necessary consequence of different databases, will also impact our ability to migrate data between Cells. -Different schemas impact our ability to reliably replicate data across Cells and especially impact our ability to validate that the data is correctly replicated. -It might force us to only be able to move data between Cells when the schemas are all in sync (slowing down deployments and the rebalancing process) or possibly only migrate from newer to older schemas which would be complex. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -### 3.1. Split large Cells - -A single Cell can only be divided into many Cells. -This is based on the principle that it is easier to create an exact clone of an existing Cell in many replicas out of which some will be made authoritative once migrated. -Keeping those replicas up-to-date with Cell 0 is also much easier due to pre-existing replication solutions that can replicate the whole systems: Geo, PostgreSQL physical replication, etc. - -1. All data of an Organization needs to not be divided across many Cells. -1. Split should be doable online. -1. New Cells cannot contain pre-existing data. -1. N Cells contain exact replica of Cell 0. -1. The data of Cell 0 is live replicated to as many Cells it needs to be split. -1. Once consensus is achieved between Cell 0 and N-Cells, the Organizations to be migrated away are marked as read-only cluster-wide. -1. The `routes` is updated on for all Organizations to be split to indicate an authoritative Cell holding the most recent data, like `gitlab-org` on `cell-100`. -1. The data for `gitlab-org` on Cell 0, and on other non-authoritative N-Cells are dormant and will be removed in the future. -1. All accesses to `gitlab-org` on a given Cell are validated about `cell_id` of `routes` to ensure that given Cell is authoritative to handle the data. - -#### More challenges of this proposal - -1. There is no streaming replication capability for Elasticsearch, but you could - snapshot the whole Elasticsearch index and recreate, but this takes hours. - It could be handled by pausing Elasticsearch indexing on the initial Cell during - the migration as indexing downtime is not a big issue, but this still needs - to be coordinated with the migration process. -1. Syncing Redis, Gitaly, CI Postgres, Main Postgres, registry Postgres, other - new data stores snapshots in an online system would likely lead to gaps - without a long downtime. You need to choose a sync point and at the sync - point you need to stop writes to perform the migration. The more data stores - there are to migrate at the same time the longer the write downtime for the - failover. We would also need to find a reliable place in the application to - actually block updates to all these systems with a high degree of - confidence. In the past we've only been confident by shutting down all Rails - services because any Rails process could write directly to any of these at - any time due to async workloads or other surprising code paths. -1. How to efficiently delete all the orphaned data. Locating all `ci_builds` - associated with half the Organizations would be very expensive if we have to - do joins. We haven't yet determined if we'd want to store an `organization_id` - column on every table, but this is the kind of thing it would be helpful for. - -### 3.2. Migrate Organization from an existing Cell - -This is different to split, as we intend to perform logical and selective replication of data belonging to a single Organization. -Today this type of selective replication is only implemented by Gitaly where we can migrate Git repository from a single Gitaly node to another with minimal downtime. - -In this model we would require identifying all resources belonging to a given Organization: database rows, object storage files, Git repositories, etc. and selectively copy them over to another (likely) existing Cell importing data into it. -Ideally ensuring that we can perform logical replication live of all changed data, but change similarly to split which Cell is authoritative for this Organization. - -1. It is hard to identify all resources belonging to an Organization. -1. It requires either downtime for the Organization or a robust system to identify live changes made. -1. It likely will require a full database structure analysis (more robust than Project import/export) to perform selective PostgreSQL logical replication. - -#### More challenges of this proposal - -1. Logical replication is still not performant enough to keep up with our - scale. Even if we could use logical replication we still don't have an - efficient way to filter data related to a single Organization without - joining all the way to the `organizations` table which will slow down - logical replication dramatically. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/data-migration.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-database-sequences.md b/doc/architecture/blueprints/cells/cells-feature-database-sequences.md index 2aeaaed7d64..9b426ed80a4 100644 --- a/doc/architecture/blueprints/cells/cells-feature-database-sequences.md +++ b/doc/architecture/blueprints/cells/cells-feature-database-sequences.md @@ -1,74 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Database Sequences' +redirect_to: 'impacted_features/database-sequences.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Database Sequences - -GitLab today ensures that every database row create has a unique ID, allowing to access a merge request, CI Job or Project by a known global ID. -Cells will use many distinct and not connected databases, each of them having a separate ID for most entities. -At a minimum, any ID referenced between a Cell and the shared schema will need to be unique across the cluster to avoid ambiguous references. -Further to required global IDs, it might also be desirable to retain globally unique IDs for all database rows to allow migrating resources between Cells in the future. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -These are some preliminary ideas how we can retain unique IDs across the system. - -### 3.1. UUID - -Instead of using incremental sequences, use UUID (128 bit) that is stored in the database. - -- This might break existing IDs and requires adding a UUID column for all existing tables. -- This makes all indexes larger as it requires storing 128 bit instead of 32/64 bit in index. - -### 3.2. Use Cell index encoded in ID - -Because a significant number of tables already use 64 bit ID numbers we could use MSB to encode the Cell ID: - -- This might limit the amount of Cells that can be enabled in a system, as we might decide to only allocate 1024 possible Cell numbers. -- This would make it possible to migrate IDs between Cells, because even if an entity from Cell 1 is migrated to Cell 100 this ID would still be unique. -- If resources are migrated the ID itself will not be enough to decode the Cell number and we would need a lookup table. -- This requires updating all IDs to 32 bits. - -### 3.3. Allocate sequence ranges from central place - -Each Cell might receive its own range of sequences as they are consumed from a centrally managed place. -Once a Cell consumes all IDs assigned for a given table it would be replenished and a next range would be allocated. -Ranges would be tracked to provide a faster lookup table if a random access pattern is required. - -- This might make IDs migratable between Cells, because even if an entity from Cell 1 is migrated to Cell 100 this ID would still be unique. -- If resources are migrated the ID itself will not be enough to decode the Cell number and we would need a much more robust lookup table as we could be breaking previously assigned sequence ranges. -- This does not require updating all IDs to 64 bits. -- This adds some performance penalty to all `INSERT` statements in Postgres or at least from Rails as we need to check for the sequence number and potentially wait for our range to be refreshed from the ID server. -- The available range will need to be stored and incremented in a centralized place so that concurrent transactions cannot possibly get the same value. - -### 3.4. Define only some tables to require unique IDs - -Maybe it is acceptable only for some tables to have a globally unique IDs. It could be Projects, Groups and other top-level entities. -All other tables like `merge_requests` would only offer a Cell-local ID, but when referenced outside it would rather use an IID (an ID that is monotonic in context of a given resource, like a Project). - -- This makes the ID 10000 for `merge_requests` be present on all Cells, which might be sometimes confusing regarding the uniqueness of the resource. -- This might make random access by ID (if ever needed) impossible without using a composite key, like: `project_id+merge_request_id`. -- This would require us to implement a transformation/generation of new ID if we need to migrate records to another Cell. This can lead to very difficult migration processes when these IDs are also used as foreign keys for other records being migrated. -- If IDs need to change when moving between Cells this means that any links to records by ID would no longer work even if those links included the `project_id`. -- If we plan to allow these IDs to not be unique and change the unique constraint to be based on a composite key then we'd need to update all foreign key references to be based on the composite key. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/database-sequences.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-explore.md b/doc/architecture/blueprints/cells/cells-feature-explore.md index 4eab99d63e7..95924e3d1e8 100644 --- a/doc/architecture/blueprints/cells/cells-feature-explore.md +++ b/doc/architecture/blueprints/cells/cells-feature-explore.md @@ -1,71 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Explore' +redirect_to: 'impacted_features/explore.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Explore - -Explore may not play a critical role in GitLab as it functions today, but GitLab today is not isolated. It is the isolation that makes Explore or some viable replacement necessary. - -The existing Group and Project Explore will initially be scoped to an Organization. However, there is a need for a global Explore that spans across Organizations to support the discoverability of public Groups and Projects, in particular in the context of discovering open source Projects. See user feedback [here](https://gitlab.com/gitlab-org/gitlab/-/issues/21582#note_1458298192) and [here](https://gitlab.com/gitlab-org/gitlab/-/issues/418228#note_1470045468). - -## 1. Definition - -The Explore functionality helps users in discovering Groups and Projects. Unauthenticated Users are only able to explore public Groups and Projects, authenticated Users can see all the Groups and Projects that they have access to, including private and internal Groups and Projects. - -## 2. Data flow - -## 3. Proposal - -The Explore feature problem falls under the broader umbrella of solving inter-Cell communication. [This topic warrants deeper research](index.md#can-different-cells-communicate-with-each-other). - -Below are possible directions for further investigation. - -### 3.1. Read only table mirror - -- Create a `shared_projects` table in the shared cluster-wide database. -- The model for this table is read-only. No inserts/updates/deletes are allowed. -- The table is filled with data (or a subset of data) from the Projects Cell-local table. - - The write model Project (which is Cell-local) writes to the local database. We will primarily use this model for anything Cell-local. - - This data is synchronized with `shared_projects` via a background job any time something changes. - - The data in `shared_projects` is stored normalized, so that all the information necessary to display the Project Explore is there. -- The Project Explore (as of today) is part of an instance-wide functionality, since it's not namespaced to any organizations/groups. - - This section will read data using the read model for `shared_projects`. -- Once the user clicks on a Project, they are redirected to the Cell containing the Organization. - -Downsides: - -- Need to have an explicit pattern to access instance-wide data. This however may be useful for admin functionalities too. -- The Project Explore may not be as rich in features as it is today (various filtering options, role you have on that Project, etc.). -- Extra complexity in managing CQRS. - -### 3.2 Explore scoped to an Organization - -The Project Explore and Group Explore are scoped to an Organization. - -Downsides: - -- No global discoverability of Groups and Projects. - -## 4. Evaluation - -The existing Group and Project Explore will initially be scoped to an Organization. Considering the [current usage of the Explore feature](https://gitlab.com/gitlab-data/product-analytics/-/issues/1302#note_1491215521), we deem this acceptable. Since all existing Users, Groups and Projects will initially be part of the default Organization, Groups and Projects will remain explorable and accessible as they are today. Only once existing Groups and Projects are moved out of the default Organization into different Organizations will this become a noticeable problem. Solutions to mitigate this are discussed in [issue #418228](https://gitlab.com/gitlab-org/gitlab/-/issues/418228). Ultimately, Explore could be replaced with a better search experience altogether. - -## 4.1. Pros - -- Initially the lack of discoverability will not be a problem. -- Only around [1.5% of all exisiting Users are using the Explore functionality on a monthly basis](https://gitlab.com/gitlab-data/product-analytics/-/issues/1302#note_1491215521). - -## 4.2. Cons - -- The GitLab owned top-level Groups would be some of the first to be moved into their own Organization and thus be detached from the explorability of the default Organization. +This document was moved to [another location](impacted_features/explore.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-git-access.md b/doc/architecture/blueprints/cells/cells-feature-git-access.md index 611b4db5f43..18fc2b61b1f 100644 --- a/doc/architecture/blueprints/cells/cells-feature-git-access.md +++ b/doc/architecture/blueprints/cells/cells-feature-git-access.md @@ -1,156 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Git Access' +redirect_to: 'impacted_features/git-access.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Git Access - -This document describes impact of Cells architecture on all Git access (over HTTPS and SSH) patterns providing explanation of how potentially those features should be changed to work well with Cells. - -## 1. Definition - -Git access is done throughout the application. -It can be an operation performed by the system (read Git repository) or by a user (create a new file via Web IDE, `git clone` or `git push` via command line). -The Cells architecture defines that all Git repositories will be local to the Cell, so no repository could be shared with another Cell. - -The Cells architecture will require that any Git operation can only be handled by a Cell holding the data. -It means that any operation either via Web interface, API, or GraphQL needs to be routed to the correct Cell. -It means that any `git clone` or `git push` operation can only be performed in the context of a Cell. - -## 2. Data flow - -The are various operations performed today by GitLab on a Git repository. -This describes the data flow how they behave today to better represent the impact. - -It appears that Git access does require changes only to a few endpoints that are scoped to a Project. -There appear to be different types of repositories: - -- Project: assigned to Group -- Wiki: additional repository assigned to Project -- Design: similar to Wiki, additional repository assigned to Project -- Snippet: creates a virtual Project to hold repository, likely tied to the User - -### 2.1. Git clone over HTTPS - -Execution of: `git clone` over HTTPS - -```mermaid -sequenceDiagram - User ->> Workhorse: GET /gitlab-org/gitlab.git/info/refs?service=git-upload-pack - Workhorse ->> Rails: GET /gitlab-org/gitlab.git/info/refs?service=git-upload-pack - Rails ->> Workhorse: 200 OK - Workhorse ->> Gitaly: RPC InfoRefsUploadPack - Gitaly ->> User: Response - User ->> Workhorse: POST /gitlab-org/gitlab.git/git-upload-pack - Workhorse ->> Gitaly: RPC PostUploadPackWithSidechannel - Gitaly ->> User: Response -``` - -### 2.2. Git clone over SSH - -Execution of: `git clone` over SSH - -```mermaid -sequenceDiagram - User ->> Git SSHD: ssh git@gitlab.com - Git SSHD ->> Rails: GET /api/v4/internal/authorized_keys - Rails ->> Git SSHD: 200 OK (list of accepted SSH keys) - Git SSHD ->> User: Accept SSH - User ->> Git SSHD: git clone over SSH - Git SSHD ->> Rails: POST /api/v4/internal/allowed?project=/gitlab-org/gitlab.git&service=git-upload-pack - Rails ->> Git SSHD: 200 OK - Git SSHD ->> Gitaly: RPC SSHUploadPackWithSidechannel - Gitaly ->> User: Response -``` - -### 2.3. Git push over HTTPS - -Execution of: `git push` over HTTPS - -```mermaid -sequenceDiagram - User ->> Workhorse: GET /gitlab-org/gitlab.git/info/refs?service=git-receive-pack - Workhorse ->> Rails: GET /gitlab-org/gitlab.git/info/refs?service=git-receive-pack - Rails ->> Workhorse: 200 OK - Workhorse ->> Gitaly: RPC PostReceivePack - Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111&service=git-receive-pack - Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 - Gitaly ->> User: Response -``` - -### 2.4. Git push over SSHD - -Execution of: `git clone` over SSH - -```mermaid -sequenceDiagram - User ->> Git SSHD: ssh git@gitlab.com - Git SSHD ->> Rails: GET /api/v4/internal/authorized_keys - Rails ->> Git SSHD: 200 OK (list of accepted SSH keys) - Git SSHD ->> User: Accept SSH - User ->> Git SSHD: git clone over SSH - Git SSHD ->> Rails: POST /api/v4/internal/allowed?project=/gitlab-org/gitlab.git&service=git-receive-pack - Rails ->> Git SSHD: 200 OK - Git SSHD ->> Gitaly: RPC ReceivePack - Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 - Gitaly ->> User: Response -``` - -### 2.5. Create commit via Web - -Execution of `Add CHANGELOG` to repository: - -```mermaid -sequenceDiagram - Web ->> Puma: POST /gitlab-org/gitlab/-/create/main - Puma ->> Gitaly: RPC TreeEntry - Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 - Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 - Gitaly ->> Puma: Response - Puma ->> Web: See CHANGELOG -``` - -## 3. Proposal - -The Cells stateless router proposal requires that any ambiguous path (that is not routable) will be made routable. -It means that at least the following paths will have to be updated to introduce a routable entity (Project, Group, or Organization). - -Change: - -- `/api/v4/internal/allowed` => `/api/v4/internal/projects/<gl_repository>/allowed` -- `/api/v4/internal/pre_receive` => `/api/v4/internal/projects/<gl_repository>/pre_receive` -- `/api/v4/internal/post_receive` => `/api/v4/internal/projects/<gl_repository>/post_receive` -- `/api/v4/internal/lfs_authenticate` => `/api/v4/internal/projects/<gl_repository>/lfs_authenticate` - -Where: - -- `gl_repository` can be `project-1111` (`Gitlab::GlRepository`) -- `gl_repository` in some cases might be a full path to repository as executed by GitLab Shell (`/gitlab-org/gitlab.git`) - -## 4. Evaluation - -Supporting Git repositories if a Cell can access only its own repositories does not appear to be complex. -The one major complication is supporting snippets, but this likely falls in the same category as for the approach to support a user's Personal Namespace. - -## 4.1. Pros - -1. The API used for supporting HTTPS/SSH and Hooks are well defined and can easily be made routable. - -## 4.2. Cons - -1. The sharing of repositories objects is limited to the given Cell and Gitaly node. -1. Cross-Cells forks are likely impossible to be supported (discover: How this works today across different Gitaly node). +This document was moved to [another location](impacted_features/git-access.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-gitlab-pages.md b/doc/architecture/blueprints/cells/cells-feature-gitlab-pages.md index 7e4ab785095..964423334c1 100644 --- a/doc/architecture/blueprints/cells/cells-feature-gitlab-pages.md +++ b/doc/architecture/blueprints/cells/cells-feature-gitlab-pages.md @@ -1,30 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: GitLab Pages' +redirect_to: 'impacted_features/gitlab-pages.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: GitLab Pages - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/gitlab-pages.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-global-search.md b/doc/architecture/blueprints/cells/cells-feature-global-search.md index 475db381ff5..0a2a89b2d45 100644 --- a/doc/architecture/blueprints/cells/cells-feature-global-search.md +++ b/doc/architecture/blueprints/cells/cells-feature-global-search.md @@ -1,35 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Global search' +redirect_to: 'impacted_features/global-search.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Global search - -When we introduce multiple Cells we intend to isolate all services related to those Cells. -This will include Elasticsearch which means our current global search functionality will not work. -It may be possible to implement aggregated search across all Cells, but it is unlikely to be performant to do fan-out searches across all Cells especially once you start to do pagination which requires setting the correct offset and page number for each search. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -Likely the first versions of Cells will not support global searches. -Later, we may consider if building global searches to support popular use cases is worthwhile. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/global-search.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-graphql.md b/doc/architecture/blueprints/cells/cells-feature-graphql.md index e8850dfbee3..69ce2128484 100644 --- a/doc/architecture/blueprints/cells/cells-feature-graphql.md +++ b/doc/architecture/blueprints/cells/cells-feature-graphql.md @@ -1,81 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: GraphQL' +redirect_to: 'impacted_features/graphql.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: GraphQL - -GitLab extensively uses GraphQL to perform efficient data query operations. -GraphQL due to it's nature is not directly routable. -The way GitLab uses it calls the `/api/graphql` endpoint, and only the query or mutation of the body request might define where the data can be accessed. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -There are at least two main ways to implement GraphQL in a Cells architecture. - -### 3.1. GraphQL routable by endpoint - -Change `/api/graphql` to `/api/organization/<organization>/graphql`. - -- This breaks all existing usages of `/api/graphql` endpoint because the API URI is changed. - -### 3.2. GraphQL routable by body - -As part of router parse GraphQL body to find a routable entity, like `project`. - -- This still makes the GraphQL query be executed only in context of a given Cell and not allowing the data to be merged. - -```json -# Good example -{ - project(fullPath:"gitlab-org/gitlab") { - id - description - } -} - -# Bad example, since Merge Request is not routable -{ - mergeRequest(id: 1111) { - iid - description - } -} -``` - -### 3.3. Merging GraphQL Proxy - -Implement as part of router GraphQL Proxy which can parse body and merge results from many Cells. - -- This might make pagination hard to achieve, or we might assume that we execute many queries of which results are merged across all Cells. - -```json -{ - project(fullPath:"gitlab-org/gitlab"){ - id, description - } - group(fullPath:"gitlab-com") { - id, description - } -} -``` - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/graphql.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-organizations.md b/doc/architecture/blueprints/cells/cells-feature-organizations.md index f1527b40ef4..6b589307404 100644 --- a/doc/architecture/blueprints/cells/cells-feature-organizations.md +++ b/doc/architecture/blueprints/cells/cells-feature-organizations.md @@ -1,36 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Organizations' +redirect_to: 'impacted_features/organizations.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Organizations - -One of the major designs of a Cells architecture is strong isolation between Groups. -Organizations as described by the [Organization blueprint](../organization/index.md) provides a way to have plausible UX for joining together many Groups that are isolated from the rest of the system. - -## 1. Definition - -Cells do require that all Groups and Projects of a single Organization can only be stored on a single Cell because a Cell can only access data that it holds locally and has very limited capabilities to read information from other Cells. - -Cells with Organizations do require strong isolation between Organizations. - -It will have significant implications on various user-facing features, like Todos, dropdowns allowing to select Projects, references to other issues or Projects, or any other social functions present at GitLab. -Today those functions were able to reference anything in the whole system. -With the introduction of Organizations this will be forbidden. - -This problem definition aims to answer effort and implications required to add strong isolation between Organizations to the system, including features affected and their data processing flow. -The purpose is to ensure that our solution when implemented consistently avoids data leakage between Organizations residing on a single Cell. - -## 2. Proposal - -See the [Organization blueprint](../organization/index.md). +This document was moved to [another location](impacted_features/organizations.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-personal-access-tokens.md b/doc/architecture/blueprints/cells/cells-feature-personal-access-tokens.md index 3aca9f1e116..115af11e3a6 100644 --- a/doc/architecture/blueprints/cells/cells-feature-personal-access-tokens.md +++ b/doc/architecture/blueprints/cells/cells-feature-personal-access-tokens.md @@ -1,31 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Personal Access Tokens' +redirect_to: 'impacted_features/personal-access-tokens.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Personal Access Tokens - -## 1. Definition - -Personal Access Tokens associated with a User are a way for Users to interact with the API of GitLab to perform operations. -Personal Access Tokens today are scoped to the User, and can access all Groups that a User has access to. - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/personal-access-tokens.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-personal-namespaces.md b/doc/architecture/blueprints/cells/cells-feature-personal-namespaces.md index e8f5c250a8e..6d5ec0c9dd6 100644 --- a/doc/architecture/blueprints/cells/cells-feature-personal-namespaces.md +++ b/doc/architecture/blueprints/cells/cells-feature-personal-namespaces.md @@ -1,30 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Personal Namespaces' +redirect_to: 'impacted_features/personal-namespaces.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Personal Namespaces - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/personal-namespaces.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-router-endpoints-classification.md b/doc/architecture/blueprints/cells/cells-feature-router-endpoints-classification.md index d403d6ff963..0143ac6ffd9 100644 --- a/doc/architecture/blueprints/cells/cells-feature-router-endpoints-classification.md +++ b/doc/architecture/blueprints/cells/cells-feature-router-endpoints-classification.md @@ -1,34 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Router Endpoints Classification' +redirect_to: 'impacted_features/router-endpoints-classification.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Router Endpoints Classification - -Classification of all endpoints is essential to properly route requests hitting the load balancer of a GitLab installation to a Cell that can serve it. -Each Cell should be able to decode each request and classify which Cell it belongs to. - -GitLab currently implements hundreds of endpoints. -This document tries to describe various techniques that can be implemented to allow the Rails to provide this information efficiently. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/router-endpoints-classification.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-schema-changes.md b/doc/architecture/blueprints/cells/cells-feature-schema-changes.md index dd0f6c0705c..bf78a4eae41 100644 --- a/doc/architecture/blueprints/cells/cells-feature-schema-changes.md +++ b/doc/architecture/blueprints/cells/cells-feature-schema-changes.md @@ -1,38 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Schema changes' +redirect_to: 'impacted_features/schema-changes.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Schema changes - -When we introduce multiple Cells that own their own databases this will complicate the process of making schema changes to Postgres and Elasticsearch. -Today we already need to be careful to make changes comply with our zero downtime deployments. -For example, [when removing a column we need to make changes over 3 separate deployments](../../../development/database/avoiding_downtime_in_migrations.md#dropping-columns). -We have tooling like `post_migrate` that helps with these kinds of changes to reduce the number of merge requests needed, but these will be complicated when we are dealing with deploying multiple Rails applications that will be at different versions at any one time. -This problem will be particularly tricky to solve for shared databases like our plan to share the `users` related tables among all Cells. - -A key benefit of Cells may be that it allows us to run different customers on different versions of GitLab. -We may choose to update our own Cell before all our customers giving us even more flexibility than our current canary architecture. -But doing this means that schema changes need to have even more versions of backward compatibility support which could slow down development as we need extra steps to make schema changes. - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/schema-changes.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-secrets.md b/doc/architecture/blueprints/cells/cells-feature-secrets.md index 681c229711d..1c4c79d96fc 100644 --- a/doc/architecture/blueprints/cells/cells-feature-secrets.md +++ b/doc/architecture/blueprints/cells/cells-feature-secrets.md @@ -1,43 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Secrets' +redirect_to: 'impacted_features/secrets.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Secrets - -Where possible, each Cell should have its own distinct set of secrets. -However, there will be some secrets that will be required to be the same for all Cells in the cluster. - -## 1. Definition - -GitLab has a lot of [secrets](https://docs.gitlab.com/charts/installation/secrets.html) that need to be configured. -Some secrets are for inter-component communication, for example, `GitLab Shell secret`, and used only within a Cell. -Some secrets are used for features, for example, `ci_jwt_signing_key`. - -## 2. Data flow - -## 3. Proposal - -1. Secrets used for features will need to be consistent across all Cells, so that the UX is consistent. - 1. This is especially true for the `db_key_base` secret which is used for - encrypting data at rest in the database - so that Projects that are - transferred to another Cell will continue to work. We do not want to have - to re-encrypt such rows when we move Projects/Groups between Cells. -1. Secrets which are used for intra-Cell communication only should be uniquely generated - per Cell. - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/secrets.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-snippets.md b/doc/architecture/blueprints/cells/cells-feature-snippets.md index bde0b098609..2963bbdec2c 100644 --- a/doc/architecture/blueprints/cells/cells-feature-snippets.md +++ b/doc/architecture/blueprints/cells/cells-feature-snippets.md @@ -1,56 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Snippets' +redirect_to: 'impacted_features/snippets.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Snippets - -Snippets will be scoped to an Organization. Initially it will not be possible to aggregate snippet collections across Organizations. See also [issue #416954](https://gitlab.com/gitlab-org/gitlab/-/issues/416954). - -## 1. Definition - -Two different types of snippets exist: - -- [Project snippets](../../../api/project_snippets.md). These snippets have URLs - like `/<group>/<project>/-/snippets/123` -- [Personal snippets](../../../user/snippets.md). These snippets have URLs like - `/-/snippets/123` - -Snippets are backed by a Git repository. - -## 2. Data flow - -## 3. Proposal - -### 3.1. Scoped to an organization - -Both project and personal snippets will be scoped to an Organization. - -- Project snippets URLs will remain unchanged, as the URLs are routable. -- Personal snippets URLs will need to change to be `/-/organizations/<organization>/snippets/123`, - so that the URL is routeable - -Creation of snippets will also be scoped to a User's current Organization. Because of that, we recommend renaming `personal snippets` to `organization snippets` once the Organization is rolled out. A User can create many independent snippet collections across multiple Organizations. - -## 4. Evaluation - -Snippets are scoped to an Organization because Gitaly is confined to a Cell. - -## 4.1. Pros - -- No need to have clusterwide Gitaly. - -## 4.2. Cons - -- We will break [snippet discovery](/ee/user/snippets.md#discover-snippets). -- Snippet access may become subordinate to the visibility of the Organization. +This document was moved to [another location](impacted_features/snippets.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-template.md b/doc/architecture/blueprints/cells/cells-feature-template.md index 3cece3dc99e..c75cc88f46c 100644 --- a/doc/architecture/blueprints/cells/cells-feature-template.md +++ b/doc/architecture/blueprints/cells/cells-feature-template.md @@ -1,30 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Problem A' +redirect_to: 'impacted_features/template.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: A - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/template.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-uploads.md b/doc/architecture/blueprints/cells/cells-feature-uploads.md index fdac3a9977c..eab7a8a4fcd 100644 --- a/doc/architecture/blueprints/cells/cells-feature-uploads.md +++ b/doc/architecture/blueprints/cells/cells-feature-uploads.md @@ -1,30 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Uploads' +redirect_to: 'impacted_features/uploads.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Uploads - -> TL;DR - -## 1. Definition - -## 2. Data flow - -## 3. Proposal - -## 4. Evaluation - -## 4.1. Pros - -## 4.2. Cons +This document was moved to [another location](impacted_features/uploads.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-user-profile.md b/doc/architecture/blueprints/cells/cells-feature-user-profile.md index fc02548f371..73f312f3762 100644 --- a/doc/architecture/blueprints/cells/cells-feature-user-profile.md +++ b/doc/architecture/blueprints/cells/cells-feature-user-profile.md @@ -1,52 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: User Profile' +redirect_to: 'impacted_features/user-profile.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: User Profile - -The existing User Profiles will initially be scoped to an Organization. Long-term, we should consider aggregating parts of the User activity across Organizations to enable Users a global view of their contributions. - -## 1. Definition - -Each GitLab account has a [User Profile](../../../user/profile/index.md), which contains information about the User and their GitLab activity. - -## 2. Data flow - -## 3. Proposal - -User Profiles will be scoped to an Organization. - -- Users can set a Home Organization as their main Organization. -- Users who do not exist in the database at all display a 404 not found error when trying to access their User Profile. -- User who haven't contributed to an Organization display their User Profile with an empty state. -- When displaying a User Profile empty state, if the profile has a Home Organization set to another Organization, we display a call-to-action allowing navigation to the main Organization. -- User Profile URLs will not reference the Organization and remain as: `/<username>`. We follow the same pattern as is used for `Your Work`, meaning that profiles are always seen in the context of an Organization. -- Breadcrumbs on the User Profile will present as `[Organization Name] / [Username]`. - -See [issue #411931](https://gitlab.com/gitlab-org/gitlab/-/issues/411931) for design proposals. - -## 4. Evaluation - -We expect the [majority of Users to perform most of their activity in one single Organization](../organization/index.md#data-exploration). -This is why we deem it acceptable to scope the User Profile to an Organization at first. -More discovery is necessary to understand which aspects of the current User Profile are relevant to showcase contributions in a global context. - -## 4.1. Pros - -- Viewing a User Profile scoped to an Organization allows you to focus on contributions that are most relevant to your Organization, filtering out the User's other activities. -- Existing User Profile URLs do not break. - -## 4.2. Cons - -- Users will lose the ability to display their entire activity, which may lessen the effectiveness of using their User Profile as a resume of achievements when working across multiple Organizations. +This document was moved to [another location](impacted_features/user-profile.md). diff --git a/doc/architecture/blueprints/cells/cells-feature-your-work.md b/doc/architecture/blueprints/cells/cells-feature-your-work.md index 08bb0bed709..344037f2a76 100644 --- a/doc/architecture/blueprints/cells/cells-feature-your-work.md +++ b/doc/architecture/blueprints/cells/cells-feature-your-work.md @@ -1,58 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Your Work' +redirect_to: 'impacted_features/your-work.md' +remove_date: '2023-11-17' --- -<!-- vale gitlab.FutureTense = NO --> - -This document is a work-in-progress and represents a very early state of the -Cells design. Significant aspects are not documented, though we expect to add -them in the future. This is one possible architecture for Cells, and we intend to -contrast this with alternatives before deciding which approach to implement. -This documentation will be kept even if we decide not to implement this so that -we can document the reasons for not choosing this approach. - -# Cells: Your Work - -Your Work will be scoped to an Organization. -Counts presented in the individual dashboards will relate to the selected Organization. - -## 1. Definition - -When accessing `gitlab.com/dashboard/`, users can find a [focused view of items that they have access to](../../../tutorials/left_sidebar/index.md#use-a-more-focused-view). -This overview contains dashboards relating to: - -- Projects -- Groups -- Issues -- Merge requests -- To-Do list -- Milestones -- Snippets -- Activity -- Workspaces -- Environments -- Operations -- Security - -## 2. Data flow - -## 3. Proposal - -Your Work will be scoped to an Organization, giving the user an overview of all the items they can access in the Organization they are currently viewing. - -- Issue, Merge request and To-Do list counts will refer to the selected Organization. - -## 4. Evaluation - -Scoping Your Work to an Organization makes sense in the context of the [proposed Organization navigation](https://gitlab.com/gitlab-org/gitlab/-/issues/417778). -Considering that [we expect most users to work in a single Organization](../organization/index.md#data-exploration), we deem this impact acceptable. - -## 4.1. Pros - -- Viewing Your Work scoped to an Organization allows Users to focus on content that is most relevant to their currently selected Organization. - -## 4.2. Cons - -- Users working across multiple Organizations will have to navigate to each Organization to access all of their work items. +This document was moved to [another location](impacted_features/your-work.md). diff --git a/doc/architecture/blueprints/cells/diagrams/cells-and-fulfillment.drawio.png b/doc/architecture/blueprints/cells/diagrams/cells-and-fulfillment.drawio.png Binary files differdeleted file mode 100644 index c5fff9dbca5..00000000000 --- a/doc/architecture/blueprints/cells/diagrams/cells-and-fulfillment.drawio.png +++ /dev/null diff --git a/doc/architecture/blueprints/cells/diagrams/index.md b/doc/architecture/blueprints/cells/diagrams/index.md index 77d12612819..14db888382e 100644 --- a/doc/architecture/blueprints/cells/diagrams/index.md +++ b/doc/architecture/blueprints/cells/diagrams/index.md @@ -22,7 +22,7 @@ Load the `.drawio.png` or `.drawio.svg` file directly into **draw.io**, which yo To create a diagram from a file: 1. Copy existing file and rename it. Ensure that the extension is `.drawio.png` or `.drawio.svg`. -1. Edit the diagram. +1. Edit the diagram. 1. Save the file. To create a diagram from scratch using [draw.io desktop](https://github.com/jgraph/drawio-desktop/releases): diff --git a/doc/architecture/blueprints/cells/diagrams/term-cell.drawio.png b/doc/architecture/blueprints/cells/diagrams/term-cell.drawio.png Binary files differindex 84a6d6d1745..639026c801f 100644 --- a/doc/architecture/blueprints/cells/diagrams/term-cell.drawio.png +++ b/doc/architecture/blueprints/cells/diagrams/term-cell.drawio.png diff --git a/doc/architecture/blueprints/cells/diagrams/term-cluster.drawio.png b/doc/architecture/blueprints/cells/diagrams/term-cluster.drawio.png Binary files differindex a6fd790ba5e..c5e3a0f7c71 100644 --- a/doc/architecture/blueprints/cells/diagrams/term-cluster.drawio.png +++ b/doc/architecture/blueprints/cells/diagrams/term-cluster.drawio.png diff --git a/doc/architecture/blueprints/cells/diagrams/term-organization.drawio.png b/doc/architecture/blueprints/cells/diagrams/term-organization.drawio.png Binary files differindex f1cb7cd92fe..9bfdba43309 100644 --- a/doc/architecture/blueprints/cells/diagrams/term-organization.drawio.png +++ b/doc/architecture/blueprints/cells/diagrams/term-organization.drawio.png diff --git a/doc/architecture/blueprints/cells/diagrams/term-top-level-group.drawio.png b/doc/architecture/blueprints/cells/diagrams/term-top-level-group.drawio.png Binary files differindex f5535409945..c8f6393f9fc 100644 --- a/doc/architecture/blueprints/cells/diagrams/term-top-level-group.drawio.png +++ b/doc/architecture/blueprints/cells/diagrams/term-top-level-group.drawio.png diff --git a/doc/architecture/blueprints/cells/glossary.md b/doc/architecture/blueprints/cells/glossary.md index 11a1fc5acc9..69824663867 100644 --- a/doc/architecture/blueprints/cells/glossary.md +++ b/doc/architecture/blueprints/cells/glossary.md @@ -1,106 +1,6 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Glossary' +redirect_to: 'goals.md#glossary' +remove_date: '2023-11-24' --- -# Cells: Glossary - -We use the following terms to describe components and properties of the Cells architecture. - -## Cell - -> Pod was renamed to Cell in <https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/121163> - -A Cell is a set of infrastructure components that contains multiple top-level groups that belong to different organizations. The components include both datastores (PostgreSQL, Redis etc.) and stateless services (web etc.). The infrastructure components provided within a Cell are shared among organizations and their top-level groups but not shared with other Cells. This isolation of infrastructure components means that Cells are independent from each other. - -<img src="diagrams/term-cell.drawio.png" height="200"> - -### Cell properties - -- Each cell is independent from the others -- Infrastructure components are shared by organizations and their top-level groups within a Cell -- More Cells can be provisioned to provide horizontal scalability -- A failing Cell does not lead to failure of other Cells -- Noisy neighbor effects are limited to within a Cell -- Cells are not visible to organizations; it is an implementation detail -- Cells may be located in different geographical regions (for example, EU, US, JP, UK) - -Discouraged synonyms: GitLab instance, cluster, shard - -## Cluster - -A cluster is a collection of Cells. - -<img src="diagrams/term-cluster.drawio.png" height="300"> - -### Cluster properties - -- A cluster holds cluster-wide metadata, for example Users, Routes, Settings. - -Discouraged synonyms: whale - -## Organizations - -GitLab references [Organizations in the initial set up](../../../topics/set_up_organization.md) and users can add a (free text) organization to their profile. There is no Organization entity established in the GitLab codebase. - -As part of delivering Cells, we propose the introduction of an `organization` entity. Organizations would represent billable entities or customers. - -Organizations are a known concept, present for example in [AWS](https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/core-concepts.html) and [GCP](https://cloud.google.com/resource-manager/docs/cloud-platform-resource-hierarchy#organizations). - -Organizations work under the following assumptions: - -1. Users care about what happens within their organizations. -1. Features need to work within an organization. -1. Only few features need to work across organizations. -1. Users understand that the majority of pages they view are only scoped to a single organization at a time. -1. Organizations are located on a single cell. - - - -### Organization properties - -- Top-level groups belong to organizations -- Organizations are isolated from each other by default meaning that cross-group features will only work for group that exist within a single organization -- User namespaces must not belong to an organization - -Discouraged synonyms: Billable entities, customers - -## Top-Level group - -Top-level group is the name given to the top most group of all other groups. Groups and projects are nested underneath the top-level group. - -Example: - -`https://gitlab.com/gitlab-org/gitlab/`: - -- `gitlab-org` is a `top-level group`; the root for all groups and projects of an organization -- `gitlab` is a `project`; a project of the organization. - -The top-level group has served as the defacto Organization entity. With the creation of Organization, top-level groups will be [nested underneath Organizations](https://gitlab.com/gitlab-org/gitlab/-/issues/394796). - -Over time there won't be a distinction between a top-level group and a group. All features that make Top-level groups different from groups will move to Organization. - -Discouraged synonyms: Root-level namespace - - - -### Top-level group properties - -- Top-level groups belonging to an organization are located on the same Cell -- Top-level groups can interact with other top-level groups that belong to the same organization - -## Users - -Users are available globally and not restricted to a single Cell. Users belong to a single organization, but can participate in many organizations through group and project membership with varying permissions. Inside organizations, users can create multiple top-level groups. User activity is not limited to a single organization but their contributions (for example TODOs) are only aggregated within an organization. This avoids the need for aggregating across cells. - -### User properties - -- Users are shared globally across all Cells -- Users can create multiple top-level groups -- Users can be a member of multiple top-level groups -- Users belong to one organization. See [!395736](https://gitlab.com/gitlab-org/gitlab/-/issues/395736) -- Users can be members of groups and projects in different organizations -- Users can administer organizations -- User activity is aggregated in an organization -- Every user has one personal namespace +This document was moved to [another location](goals.md#glossary). diff --git a/doc/architecture/blueprints/cells/goals.md b/doc/architecture/blueprints/cells/goals.md index 3f3923aa255..f1a255aa879 100644 --- a/doc/architecture/blueprints/cells/goals.md +++ b/doc/architecture/blueprints/cells/goals.md @@ -6,7 +6,9 @@ description: 'Cells: Goals' # Cells: Goals -## Scalability +## Goals + +### Scalability The main goal of this new shared-infrastructure architecture is to provide additional scalability for our SaaS Platform. GitLab.com is largely monolithic and we have estimated (internally) that the current architecture has scalability limitations, @@ -16,34 +18,41 @@ even when database partitioning and decomposition are taken into account. Cells provide a horizontally scalable solution because additional Cells can be created based on demand. Cells can be provisioned and tuned as needed for optimal scalability. -## Increased availability +### Increased availability -A major challenge for shared-infrastructure architectures is a lack of isolation between top-level groups. This can lead to noisy neighbor effects. A organization's behavior inside a top-level group can impact all other organizations. This is highly undesirable. Cells provide isolation at the cell level. A group of organizations is fully isolated from other organizations located on a different Cell. This minimizes noisy neighbor effects while still benefiting from the cost-efficiency of shared infrastructure. +A major challenge for shared-infrastructure architectures is a lack of isolation between top-level Groups. +This can lead to noisy neighbor effects. +An organization's behavior inside a top-level Group can impact all other organizations. +This is highly undesirable. +Cells provide isolation at the Cell level. +A group of Organizations is fully isolated from other Organizations located on a different Cell. +This minimizes noisy neighbor effects while still benefiting from the cost-efficiency of a shared infrastructure. -Additionally, Cells provide a way to implement disaster recovery capabilities. Entire Cells may be replicated to read-only standbys with automatic failover capabilities. +Additionally, Cells provide a way to implement disaster recovery capabilities. +Entire Cells may be replicated to read-only standbys with automatic failover capabilities. -## A consistent experience +### A consistent experience Organizations should have the same user experience on our SaaS platform as they do on a self-managed GitLab instance. -## Regions - -GitLab.com is only hosted within the United States of America. Organizations located in other regions have voiced demand for local SaaS offerings. Cells provide a path towards [GitLab Regions](https://gitlab.com/groups/gitlab-org/-/epics/6037) because Cells may be deployed within different geographies. Depending on which of the organization's data is located outside a Cell, this may solve data residency and compliance problems. +### Regions -## Market segment +GitLab.com is only hosted within the United States of America. +Organizations located in other regions have voiced demand for local SaaS offerings. +Cells provide a path towards [GitLab Regions](https://gitlab.com/groups/gitlab-org/-/epics/6037) because Cells may be deployed within different geographies. +Depending on which of an organization's data is located outside a Cell, this may solve data residency and compliance problems. -Cells would provide a solution for organizations in the small to medium business (up to 100 users) and the mid-market segment (up to 2000 users). -(See [segmentation definitions](https://about.gitlab.com/handbook/sales/field-operations/gtm-resources/#segmentation).) -Larger organizations may benefit substantially from [GitLab Dedicated](../../../subscriptions/gitlab_dedicated/index.md). +### Market segment -At this moment, GitLab.com has "social-network"-like capabilities that may not fit well into a more isolated organization model. Removing those features, however, possesses some challenges: +At this moment, GitLab.com has "social network"-like capabilities that may not fit well into a more isolated Organization model. +Removing those features, however, poses some challenges: -1. How will existing `gitlab-org` contributors contribute to the namespace?? -1. How do we move existing top-level groups into the new model (effectively breaking their social features)? +1. How will existing `gitlab-org` contributors contribute to the namespace? +1. How do we move existing top-level Groups into the new model (effectively breaking their social features)? -We should evaluate if the SMB and mid market segment is interested in these features, or if not having them is acceptable in most cases. +We should evaluate if the small to medium business and mid-market segment is interested in these features, or if not having them is acceptable in most cases. -## Self-managed +### Self-managed For reasons of consistency, it is expected that self-managed instances will adopt the cells architecture as well. To expand, self-managed instances can @@ -51,13 +60,341 @@ continue with just a single Cell while supporting the option of adding additiona Cells. Organizations, and possible User decomposition will also be adopted for self-managed instances. -## High-level architecture problems to solve +## Requirements + +| Type | Requirement | Severity | +| ----------- | ------------------------------------------------- | -------- | +| Product | Aggregation of cluster-wide data | Medium | +| Product | All Cells are under a single GitLab.com domain | High | +| Product | User can interact with many Cells | Medium | +| Product | Minimal impact on contributor workflows | High | +| Product | On-premise like experience | Medium | +| Product | Minimize breaking changes | High | +| Product | Enables support for Regions | Low | +| Product | Limit impact on self-managed customers | Low | +| Operational | Provides 10x headroom | High | +| Operational | Provides 100x headroom | Medium | +| Operational | Improve Service Availability | High | +| Operational | Cost per user similar or lower to GitLab.com | Medium | +| Operational | Unified way of deploying Cells | Medium | +| Operational | Cells running in mixed deployments | High | +| Operational | High resilience to a single Cell failure | High | +| Operational | Small Cells | Medium | +| Migration | Robust rollback and disaster recovery scenarios | High | +| Migration | Scale existing GitLab.com database | High | +| Migration | Re-balancing of Cells | Low | +| Migration | Customer can migrate from GitLab.com to Dedicated | Low | +| Development | Easy usage in development environment | Medium | + +### Aggregation of cluster-wide data + +The architecture should provide a way to present the cluster of Cells in a single view. +This might mean to: + +- Aggregate user To-Dos from different Organizations +- Perform cluster-wide searches of public Projects, or published source code + +### All Cells are under a single GitLab.com domain + +General users should be unaware of the presence of Cells. +Cells would only be visible to instance administrators. +Users using Organizations on another Cell or us migrating Organizations +between Cells should be an operational detail that does not require +any user intervention or is not visible to the user. + +### User can interact with many Cells + +Usage of many accounts to interact with different Organizations +that might be on different Cells is strongly discouraged. +The need to use many accounts will reduce adoption and put a barrier +on users wanting to contribute to open-source Projects. + +In the future, users could have Organization specific settings, +depending on the specific customer expectations. +For instance, users might be shown with different display names in different Organizations. + +Users should not be required to use different SSH keys to access different +Organizations due to the complexity of managing many SSH keys by the user. + +### Minimal impact on contributor workflows + +GitLab.com has a number of open-source and open-core projects (including [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab)). +Cells should not make it harder to contribute to public Projects. +Learning new ways to contribute is likely to hinder the adoption of Cells. +The introduced architecture should focus on changing the existing workflows in the most minimal way. + +### On-premise like experience + +Currently on-premise has numerous advantages compared to our SaaS offering. +On-premise allows to control all aspects of a GitLab installation, including but not limited to: managing users, access controls, or instance-wide settings. + +The difference between SaaS and on-premise is problematic for our customers, +and us as it results in a different user experience. + +### Minimize breaking changes + +The introduction of Cells will imply changes in the supported GitLab.com workflows. +Cells might affect how people use GitLab, as long as it provides a compelling user experience story. +It is desired that each change introduced provides either operational value or user value, ideally both. + +Any Cells architecture should focus on not introducing breaking changes, +or if introducing them provide a long window to support the change. +The introduction of breaking changes will reduce the adoption rate +and effectiveness of Cells and make rollout of the architecture more risky. + +It is important to distinguish between different severity of breaking changes: + +- Required: migration to a new architecture by design introduces a breaking change. + In such a case a customer has to use a new workflow, new API, or new endpoints right away + and is required to adapt to all changes at once. The customer has limited options to roll back. + +- Optional: migration to a new architecture does not enforce introducing breaking changes. + In such a case a customer can slowly adapt to introduced changes: a new workflow, new API, + or new endpoints, but retaining major aspects of the system to work as before. + The customer has a way to roll back to a previous known working state. + +### Enables support for Regions + +Support for Regions should enable us to run GitLab in different availability zones, +or completely different data centers. + +This includes, but is not limited to allowing users to create Organizations in Europe, the US West or US East, +and GitLab Inc. to use different cloud infrastructure providers (Google Cloud, AWS) to serve customers. + +Cells enable existing customers running Organizations to move their Organization between different deployment regions/data centers that are supported by GitLab on GitLab.com. + +### Limit impact on self-managed customers (Omnibus/CNG) + +The introduction of Cells should not impact small installations. +Cells should not require running additional components by self-managed customers +unless they are interested in Cells that will increase resource requirements. + +### Provides 10x headroom + +The Cells architecture must provide at least 10x headroom. As such our architecture must be suitable to run 10 Cells. We do not need to initially solve for the complexities that might come with running more than 10 Cells. + +### Provides 100x headroom + +The Cells architecture should be able to run more than 10 Cells. + +### Improve Service Availability + +The Cells architecture should allow us to provide better SLA for some customers +by putting them on specific Cells of the cluster: + +- Cells have to reduce the impact of noisy neighbors, such as legitimate usage caused by a spiky workload. +- Cells have to reduce the impact caused by abuse, for instance CI being used for cryptocurrency mining. + +### Cost per user similar or lower to GitLab.com + +The current GitLab.com architecture is rather cost-effective. +The introduction of Cells will result in more infrastructure components +due to data distribution: + +- The proposed Cells architecture should evaluate the cost of running additional Cells with respect + to the achieved infrastructure component isolation. In some cases it might be beneficial to reuse + infrastructure components to reduce the running cost of Cells. +- The proposed Cells architecture should ensure a high degree of multi-tenancy on Cells. +- The proposed Cells architecture could enable a way to balance the Cells load long term. + +### Unified way of deploying Cells + +The proposed Cells architecture anticipates a need to run 100 Cells and more in a cluster. +This becomes operational overhead. It is strongly desired for the Cells architecture +to reuse existing infrastructure tooling as much as possible for: deployment, monitoring, +logging, disaster recovery, and customer support. + +### Cells running in mixed deployments + +It is strongly required for the Cells architecture to run different versions of applications across the cluster. The purpose is to provide staged deployments allowing to test changes on part of the cluster, and to update part of the cluster less frequently. + +This also indicates that Cells should be able to work with different versions of the underlying +infrastructure components: PostgreSQL version, Redis version, Gitaly version, Elasticsearch version. + +Usage of different versions within a Cell has to have no effect on other Cells. + +### High resilience to a single Cell failure + +Unavailability of a single Cell should not result in other Cells being unable to function properly: + +- Intra-cluster communication between Cells should be resilient to single Cell failure. +- Intra-cluster communication should limit its dependence on data stored in other Cells. +- Cluster-shared data should provide an anti-corruption layer to prevent a single Cell from causing a cluster-wide outage. +- Cells should be monitored and their availability status should be accessible to all other Cells. +- Cluster-wide data aggregation should take the availability status of each Cell into account. +- The loss of cell-local data (groups and projects) does only affect ability to access data located on this cell. + +### Small Cells + +The Cells architecture should provide a cost effective way to run small Cells. +Running small Cells allows to achieve superior quality in terms of latency or performance +due to processing a significantly smaller data set and allowing significant single-node vertical scaling. + +The ability to run small Cells could result in reduced complexity of individual Cell deployment: + +- Prefer single-node vertical scaling as a first technique for increasing capacity (increase CPU count, available memory). +- Reduce amount of data stored within a Cell to reduce time-to-recovery in case of outages. +- Less data results in faster database migrations, improved latency, and superior user-facing performance. + +### Robust rollback and disaster recovery scenarios + +Rollout of the Cells architecture is a fundamental change to how GitLab.com will operate. +This will involve introducing a number of new components, and distributing data horizontally across +different Cells. + +We anticipate that something will go wrong at some point. Each deployment phase of the Cells architecture needs to provide a way to roll back to a prior working state. This includes any user-facing changes, as well as any infrastructure deployed components. + +### Scale existing GitLab.com database + +The existing GitLab.com database will continue to grow while we deploy Cells. +The Cells architecture should provide a way to increase Cell 1 (existing GitLab.com) +capacity as soon as possible. + +### Re-balancing of Cells + +It is expected that data distribution on Cells will become uneven at some point. + +The problem of re-balancing Cells is rather complex, but at some scale it might +provide a positive ROI compared to the running cost of imbalanced Cells. + +This might involve understanding how to selectively migrate customer data between +Cells onto already existing Cells. + +### Customer can migrate from GitLab.com to Dedicated + +Cells will enforce an architecture with more isolation. However, it is expected +that some customers would like to move from SaaS GitLab.com to GitLab Dedicated. + +This implies being able to selectively migrate data from the GitLab.com cluster to Dedicated. +This could be achieved by: + +- Migrating a customer to a Dedicated Cell on the GitLab.com cluster. +- Disconnecting a Cell from the GitLab.com cluster into its own standalone Dedicated Cell. + +### Easy usage in development environment + +The Cells architecture should be easy to run locally by developers as needed to be able to: + +- Implement cluster-wide features. +- Model Cells not being available. + +## Non-Goals + +The following objectives are not part of this document. At some point they +were considered, but dismissed as creating distraction from the goals and requirements documented above. + +### Federation between distinct GitLab instances + +The Cells architecture is meant to provide trusted intra-cluster communication +with some set of data being shared. Federation is meant to solve the problem +of data flow between two completely distinct and external instances. + +### Usage of cloud managed services + +Cells should not interfere with other efforts to move towards using more managed services in the cloud. + +### Cells should replace canary deployments + +When we have Cells we won't need to run canary deployments anymore as we can roll out changes to single Cells at a time. +Our main benefit of canary deployments today is that we can roll out code changes to internal users first. +With Cells, we will have internal users on a specific Cell, which can be deployed first. + +Additionally Cells may solve some issues with canaries today, for example not having a way to upgrade Sidekiq code for a subset of users. + +## Glossary + +| Term | Description | Discouraged Terms | +| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | +| [Cell](#cell) | A Cell is a set of infrastructure components that contains multiple top-level Groups that belong to different Organizations. | GitLab instance, cluster, shard, Pod | +| [Cluster](#cluster) | A cluster is a collection of Cells. | Whale, GitLab Dedicated instance, instance | +| [Organizations](#organizations) | An Organization is the umbrella for one or multiple top-level Groups. | Billable entities, customers | +| [Top-Level Group](#top-level-group) | Top-level Group is the name given to the top-most Group of all other Groups. Groups and Projects are nested underneath the top-level Group. | Root-level namespace | +| [Users](#users) | An account containing its own personal namespace associated with an email. | Customer | + +### Cell + +> Pod was renamed to Cell in <https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/121163> + +A Cell is a set of infrastructure components that contains multiple top-level Groups that belong to different Organizations. The components include both data stores (PostgreSQL, Redis, etc.) and stateless services (web, etc.). The infrastructure components provided within a Cell are shared among Organizations and their top-level Groups, but are not shared with other Cells. This isolation of infrastructure components means that Cells are independent from each other. + +<img src="diagrams/term-cell.drawio.png" height="200"> + +- Each Cell is independent from other Cells +- Infrastructure components are shared by Organizations and their top-level Groups within a Cell +- More Cells can be provisioned to provide horizontal scalability +- A failing Cell does not lead to failure of other Cells +- Noisy neighbor effects are limited to within a Cell +- Cells are not visible to Organizations - they are an implementation detail +- Cells may be located in different geographical regions (for example: EU, US, JP, UK) +- A GitLab Dedicated instance can join the cluster, and become a Cell + +Discouraged synonyms: GitLab instance, cluster, shard, Pod + +### Cluster + +A cluster is a collection of Cells. + +<img src="diagrams/term-cluster.drawio.png" height="300"> + +- A cluster holds cluster-wide metadata, for example: Users, Routes, Settings. + +Discouraged synonyms: Whale, GitLab Dedicated instance, instance + +### Organizations + +An Organization is the umbrella for one or multiple top-level Groups. Organizations are isolated from each other by default meaning that cross-namespace features will only work for namespaces that exist in a single Organization. + + + +See the [Organization blueprint](../../blueprints/organization/index.md). + +Organizations are a known concept, present for example in [AWS](https://docs.aws.amazon.com/whitepapers/latest/organizing-your-aws-environment/core-concepts.html) and [GCP](https://cloud.google.com/resource-manager/docs/cloud-platform-resource-hierarchy#organizations). + +Organizations work under the following assumptions: + +1. Users care about what happens within their Organization. +1. Features need to work within an Organization. +1. Only few features need to work across Organizations. +1. Users understand that the majority of pages they view are only scoped to a single Organization at a time. +1. Organizations are located on a single Cell. + +Properties: + +- Top-level Groups belong to Organizations. +- Organizations are isolated from each other by default, meaning that cross-Group features will only work for Groups that exist within a single Organization. +- Personal namespaces must not belong to an Organization. + +Discouraged synonyms: Billable entities, customers + +### Top-level Group + +Example: + +`https://gitlab.com/gitlab-org/gitlab/`: + +- `gitlab-org` is a `top-level group`; the root for all Groups and Projects of an Organization +- `gitlab` is a `project`; a Project of the Organization. + +The top-level Group has served as the de facto Organization entity. With the creation of Organizations, top-level Groups will be [nested underneath Organizations](https://gitlab.com/gitlab-org/gitlab/-/issues/394796). + +Over time, there won't be a distinction between a top-level Group and a Group. All features that make top-level Groups different from Groups will move to the Organization. + + + +- Top-level Groups belonging to the same Organization are located on the same Cell. +- Top-level Groups can interact with other top-level Groups that belong to the same Organization. + +Discouraged synonyms: Root-level namespace + +### Users -A number of technical issues need to be resolved to implement Cells (in no particular order). This section will be expanded. +Users are available globally and not restricted to a single Cell. Users belong to a single Organization, but can participate in many Organizations through Group and Project membership with varying permissions. Inside Organizations, users can create multiple top-level Groups. User activity is not limited to a single Organization but their contributions (for example To-Dos) are only aggregated within an Organization. This avoids the need for aggregating across Cells. -1. How are Cells provisioned? - [Design discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/396641) -1. What is a Cells topology? - [Design discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/396641) -1. How are users of an organization routed to the correct Cell? - -1. How do users authenticate with Cells and Organizations? - [Design discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/395736) -1. How are Cells rebalanced? -1. How can Cells implement disaster recovery capabilities? +- Users are shared globally across all Cells. +- Users can create multiple top-level Groups. +- Users can be members of multiple top-level Groups. +- Users belong to one Organization. See [!395736](https://gitlab.com/gitlab-org/gitlab/-/issues/395736). +- Users can be members of Groups and Projects in different Organizations. +- Users can administer Organizations. +- User activity is aggregated in an Organization. +- Every user has one personal namespace. diff --git a/doc/architecture/blueprints/cells/impact.md b/doc/architecture/blueprints/cells/impact.md index 30c70dca0cc..1f77b9056be 100644 --- a/doc/architecture/blueprints/cells/impact.md +++ b/doc/architecture/blueprints/cells/impact.md @@ -1,58 +1,7 @@ --- -stage: enablement -group: Tenant Scale -description: 'Cells: Cross-section impact' +redirect_to: 'index.md' +remove_date: '2023-11-22' --- -# Cells: Cross-section impact - -Cells is a fundamental architecture change that impacts other sections and stages. This section summarizes and links to other groups that may be impacted and highlights potential conflicts that need to be resolved. The Tenant Scale group is not responsible for achieving the goals of other groups but we want to ensure that dependencies are resolved. - -## Summary - -Based on discussions with other groups the net impact of introducing Cells and a new entity called organizations is mostly neutral. It may slow down development in some areas. We did not discover major blockers for other teams. - -1. We need to resolve naming conflicts (proposal is TBD) -1. Cells requires introducing Organizations. Organizations are a new entity **above** top-level groups. Because this is a new entity, it may impact the ability to consolidate settings for Group::Organization and influence their decision on [how to approach introducing a an organization](https://gitlab.com/gitlab-org/gitlab/-/issues/376285#approach-2-organization-is-built-on-top-of-top-level-groups) -1. Organizations may make it slightly easier for Fulfillment to realize their billing plans. - -## Impact on Group::Organization - -We synced with the Organization PM and Designer ([recording](https://youtu.be/b5Opn9cFWFk)) and discussed the similarities and differences between the Cells and Organization proposal ([presentation](https://docs.google.com/presentation/d/1FsUi22Up15b_tu6p2m-yLML3hCZ3rgrZrmzJAxUsNmU/edit?usp=sharing)). - -### Goals of Group::Organization - -As defined in the [organization documentation](../../../user/organization/index.md): - -1. Create an entity to manage everything you do as a GitLab administrator, including: - 1. Defining and applying settings to all of your groups, subgroups, and projects. - 1. Aggregating data from all your groups, subgroups, and projects. -1. Reach feature parity between SaaS and self-managed installations, with all Admin Area settings moving to groups (?). Hardware controls remain on the instance level. - -The [organization roadmap outlines](https://gitlab.com/gitlab-org/gitlab/-/issues/368237#high-level-goals) the current goals in detail. - -### Potential conflicts with Cells - -- Organization defines a new entity as the primary organizational object for groups and projects. -- We will only introduce one entity -- Group::Organization highlighted the need to further validate the key assumption that users only care about what happens within their organization. - -## Impact on Fulfillment - -We synced with Fulfillment ([recording](https://youtu.be/FkQF3uF7vTY)) to discuss how Cells would impact them. Fulfillment is supportive of an entity above top-level groups. Their perspective is outline in [!5639](https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/5639/diffs). - -### Goals of Fulfillment - -- Fulfillment has a longstanding plan to move billing from the top-level group to a level above. This would mean that a license applies for an organization and all its top-level groups. -- Fulfillment uses Zuora for billing and would like to have a 1-to-1 relationship between an organization and their Zuora entity called BillingAccount. They want to move away from tying a license to a single user. -- If a customer needs multiple organizations, the corresponding BillingAccounts can be rolled up into a consolidated billing account (similar to [AWS consolidated billing](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/consolidated-billing.html)) -- Ideally, a self-managed instance has a single Organization by default, which should be enough for most customers. -- Fulfillment prefers only one additional entity. - -A rough representation of this is: - - - -### Potential conflicts with Cells - -- There are no known conflicts between Fulfillment's plans and Cells +This document was removed due to being outdated. +Go to [index page](index.md) for the most recent content. diff --git a/doc/architecture/blueprints/cells/impacted_features/admin-area.md b/doc/architecture/blueprints/cells/impacted_features/admin-area.md new file mode 100644 index 00000000000..a9cd170b2a7 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/admin-area.md @@ -0,0 +1,81 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Admin Area' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Admin Area + +In our Cells architecture proposal we plan to share all admin related tables in GitLab. +This allows for simpler management of all Cells in one interface and reduces the risk of settings diverging in different Cells. +This introduces challenges with Admin Area pages that allow you to manage data that will be spread across all Cells. + +## 1. Definition + +There are consequences for Admin Area pages that contain data that span "the whole instance" as the Admin Area pages may be served by any Cell or possibly just one Cell. +There are already many parts of the Admin Area that will have data that span many Cells. +For example lists of all Groups, Projects, Topics, Jobs, Analytics, Applications and more. +There are also administrative monitoring capabilities in the Admin Area that will span many Cells such as the "Background Jobs" and "Background Migrations" pages. + +## 2. Data flow + +## 3. Proposal + +We will need to decide how to handle these exceptions with a few possible +options: + +1. Move all these pages out into a dedicated per-Cell admin section. Probably + the URL will need to be routable to a single Cell like `/cells/<cell_id>/admin`, + then we can display these data per Cell. These pages will be distinct from + other Admin Area pages which control settings that are shared across all Cells. We + will also need to consider how this impacts self-managed customers and + whether, or not, this should be visible for single-Cell instances of GitLab. +1. Build some aggregation interfaces for this data so that it can be fetched + from all Cells and presented in a single UI. This may be beneficial to an + administrator that needs to see and filter all data at a glance, especially + when they don't know which Cell the data is on. The downside, however, is + that building this kind of aggregation is very tricky when all Cells are + designed to be totally independent, and it does also enforce stricter + requirements on compatibility between Cells. + +The following overview describes at what level each feature contained in the current Admin Area will be managed: + +| Feature | Cluster | Cell | Organization | +| --- | --- | --- | --- | +| Abuse reports | | | | +| Analytics | | | | +| Applications | | | | +| Deploy keys | | | | +| Labels | | | | +| Messages | ✓ | | | +| Monitoring | | ✓ | | +| Subscription | | | | +| System hooks | | | | +| Overview | | | | +| Settings - General | ✓ | | | +| Settings - Integrations | ✓ | | | +| Settings - Repository | ✓ | | | +| Settings - CI/CD (1) | ✓ | ✓ | | +| Settings - Reporting | ✓ | | | +| Settings - Metrics | ✓ | | | +| Settings - Service usage data | | ✓ | | +| Settings - Network | ✓ | | | +| Settings - Appearance | ✓ | | | +| Settings - Preferences | ✓ | | | + +(1) Depending on the specific setting, some will be managed at the cluster-level, and some at the Cell-level. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/agent-for-kubernetes.md b/doc/architecture/blueprints/cells/impacted_features/agent-for-kubernetes.md new file mode 100644 index 00000000000..37347cf836d --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/agent-for-kubernetes.md @@ -0,0 +1,30 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Agent for Kubernetes' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Agent for Kubernetes + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/backups.md b/doc/architecture/blueprints/cells/impacted_features/backups.md new file mode 100644 index 00000000000..3d20d6e2caa --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/backups.md @@ -0,0 +1,53 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Backups' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Backups + +Each Cell will take its own backups, and consequently have its own isolated backup/restore procedure. + +## 1. Definition + +GitLab backup takes a backup of the PostgreSQL database used by the application, and also Git repository data. + +## 2. Data flow + +Each Cell has a number of application databases to back up (for example, `main`, and `ci`). +Additionally, there may be cluster-wide metadata tables (for example, `users` table) which is directly accessible via PostgreSQL. + +## 3. Proposal + +### 3.1. Cluster-wide metadata + +It is currently unknown how cluster-wide metadata tables will be accessible. +We may choose to have cluster-wide metadata tables backed up separately, or have each Cell back up its copy of cluster-wide metadata tables. + +### 3.2 Consistency + +#### 3.2.1 Take backups independently + +As each Cell will communicate with each other via API, and there will be no joins to the `users` table, it should be acceptable for each Cell to take a backup independently of each other. + +#### 3.2.2 Enforce snapshots + +We can require that each Cell take a snapshot for the PostgreSQL databases at around the same time to allow for a consistent enough backup. + +## 4. Evaluation + +As the number of Cells increases, it will likely not be feasible to take a snapshot at the same time for all Cells. +Hence taking backups independently is the better option. + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/ci-cd-catalog.md b/doc/architecture/blueprints/cells/impacted_features/ci-cd-catalog.md new file mode 100644 index 00000000000..3ca2ff042dc --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/ci-cd-catalog.md @@ -0,0 +1,54 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: CI/CD Catalog' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the Cells design. +Significant aspects are not documented, though we expect to add them in the future. +This is one possible architecture for Cells, and we intend to contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that we can document the reasons for not choosing this approach. + +# Cells: CI/CD Catalog + +The [CI/CD pipeline components catalog](../../ci_pipeline_components/index.md) is a currently experimental feature that aims at helping users reuse pipeline configurations. +Potentially, there are several aspects of the CI/CD catalog that might be affected by Cells: + +1. Namespace catalog, exists today as an experimental feature. With the introduction of Cells we would likely remove the namespace catalog and create a single Organization catalog, where all users would see all available published components in a single place (based on their permissions). This would replace today's offering, where users can have multiple catalogs which are bound to a namespace and completely isolated from each other. +1. The community catalog is supposed to allow users to search across different Organizations. + +## 1. Definition + +The [CI/CD pipeline components catalog](../../ci_pipeline_components/index.md) makes reusing pipeline configurations easier and more efficient. +It provides a way to discover and reuse pipeline constructs, allowing for a more streamlined experience. + +There are several flavors of the CI/CD catalog: + +1. [Namespace catalog (experimental)](../../../../ci/components/index.md): Bound to the top-level namespace (group or personal namespace). The namespace catalog aggregates all published components from Projects it contains. The number of top-level namespaces available in an Organization could potentially be the number of available catalogs. +1. Instance-wide component catalog (planned): Surfacing all the components that are scattered across an instance. All published components in a public or internal Project will be available in the instance-wide catalog. Only a single instance-wide catalog is planned per instance. +1. Community catalog (planned): Allow users to search all published components in different repositories across multiple namespaces. The original plan was to introduce a community catalog within self-managed customer that would act as an aggregator of all published components hosted in that instance. + +## 2. Data flow + +## 3. Proposal + +Moving to Organizations is a great opportunity to improve the user experience and to reach parity for both self-managed and GitLab.com users. + +- We introduce an Organization catalog which aggregates and surfaces all the published components that are hosted in a single Organization. The Organization catalog would make the namespace catalog obsolete. +- Once Organizations exist, GitLab.com users would need a community catalog to surface components across multiple Organizations. We need additional research to understand if such a solution is needed for self-managed customers as well. + +## 4. Evaluation + +Moving to a single Organization will improve the experience for users of the CI/CD component catalog. +Today we can have multiple catalogs based on the number of namespaces, making it difficult for users to surface information across an Organization. + +### 4.1. Pros + +- An Organization catalog will be one unified catalog serving as the single source of truth for an Organization. +- An Organization catalog would serve both self-managed and GitLab.com users, whereas the current plan was to introduce two types of catalogs: an instance-wide component catalog for self-managed and a community catalog for GitLab.com. + +### 4.2. Cons + +- A separate catalog that surfaces components across Organizations would need to be implemented to serve the wider community (community catalog). This catalog will be required for GitLab.com only, later on we can evaluate if a similar catalog is needed for self-managed customers. diff --git a/doc/architecture/blueprints/cells/impacted_features/ci-runners.md b/doc/architecture/blueprints/cells/impacted_features/ci-runners.md new file mode 100644 index 00000000000..e9fe4905573 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/ci-runners.md @@ -0,0 +1,144 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: CI Runners' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: CI Runners + +GitLab executes CI jobs via [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/), very often managed by customers in their infrastructure. +All CI jobs created as part of the CI pipeline are run in the context of a Project. +This poses a challenge how to manage GitLab Runners. + +## 1. Definition + +There are 3 different types of runners: + +- Instance-wide: Runners that are registered globally with specific tags (selection criteria) +- Group runners: Runners that execute jobs from a given top-level Group or Projects in that Group +- Project runners: Runners that execute jobs from one Projects or many Projects: some runners might + have Projects assigned from Projects in different top-level Groups. + +This, alongside with the existing data structure where `ci_runners` is a table describing all types of runners, poses a challenge as to how the `ci_runners` should be managed in a Cells environment. + +## 2. Data flow + +GitLab runners use a set of globally scoped endpoints to: + +- Register a new runner via registration token `https://gitlab.com/api/v4/runners` + ([subject for removal](../../runner_tokens/index.md)) (`registration token`) +- Create a new runner in the context of a user `https://gitlab.com/api/v4/user/runners` (`runner token`) +- Request jobs via an authenticated `https://gitlab.com/api/v4/jobs/request` endpoint (`runner token`) +- Upload job status via `https://gitlab.com/api/v4/jobs/:job_id` (`build token`) +- Upload trace via `https://gitlab.com/api/v4/jobs/:job_id/trace` (`build token`) +- Download and upload artifacts via `https://gitlab.com/api/v4/jobs/:job_id/artifacts` (`build token`) + +Currently three types of authentication tokens are used: + +- Runner registration token ([subject for removal](../../runner_tokens/index.md)) +- Runner token representing a registered runner in a system with specific configuration (`tags`, `locked`, etc.) +- Build token representing an ephemeral token giving limited access to updating a specific job, uploading artifacts, downloading dependent artifacts, downloading and uploading container registry images + +Each of those endpoints receive an authentication token via header (`JOB-TOKEN` for `/trace`) or body parameter (`token` all other endpoints). + +Since the CI pipeline would be created in the context of a specific Cell, it would be required that pick of a build would have to be processed by that particular Cell. +This requires that build picking depending on a solution would have to be either: + +- Routed to the correct Cell for the first time +- Be two-phased: Request build from global pool, claim build on a specific Cell using a Cell specific URL + +## 3. Proposal + +### 3.1. Authentication tokens + +Even though the paths for CI runners are not routable, they can be made routable with these two possible solutions: + +- The `https://gitlab.com/api/v4/jobs/request` uses a long polling mechanism with + a ticketing mechanism (based on `X-GitLab-Last-Update` header). When the runner first + starts, it sends a request to GitLab to which GitLab responds with either a build to pick + by runner. This value is completely controlled by GitLab. This allows GitLab + to use JWT or any other means to encode a `cell` identifier that could be easily + decodable by Router. +- The majority of communication (in terms of volume) is using `build token`, making it + the easiest target to change since GitLab is the sole owner of the token that the runner later + uses for a specific job. There were prior discussions about not storing the `build token` + but rather using a `JWT` token with defined scopes. Such a token could encode the `cell` + to which the Router could route all requests. + +### 3.2. Request body + +- The most used endpoints pass the authentication token in the request body. It might be desired + to use HTTP headers as an easier way to access this information by Router without + a need to proxy requests. + +### 3.3. Instance-wide are Cell-local + +We can pick a design where all runners are always registered and local to a given Cell: + +- Each Cell has its own set of instance-wide runners that are updated at its own pace +- The Project runners can only be linked to Projects from the same Organization, creating strong isolation. +- In this model the `ci_runners` table is local to the Cell. +- In this model we would require the above endpoints to be scoped to a Cell in some way, or be made routable. It might be via prefixing them, adding additional Cell parameters, or providing much more robust ways to decode runner tokens and match it to a Cell. +- If a routable token is used, we could move away from cryptographic random stored in database to rather prefer to use JWT tokens. +- The Admin Area showing registered runners would have to be scoped to a Cell. + +This model might be desired because it provides strong isolation guarantees. +This model does significantly increase maintenance overhead because each Cell is managed separately. +This model may require adjustments to the runner tags feature so that Projects have a consistent runner experience across Cells. + +### 3.4. Instance-wide are cluster-wide + +Contrary to the proposal where all runners are Cell-local, we can consider that runners +are global, or just instance-wide runners are global. + +However, this requires significant overhaul of the system and we would have to change the following aspects: + +- The `ci_runners` table would likely have to be decomposed into `ci_instance_runners`, ... +- All interfaces would have to be adopted to use the correct table. +- Build queuing would have to be reworked to be two-phased where each Cell would know of all pending and running builds, but the actual claim of a build would happen against a Cell containing data. +- It is likely that `ci_pending_builds` and `ci_running_builds` would have to be made `cluster-wide` tables, increasing the likelihood of creating hotspots in a system related to CI queueing. + +This model is complex to implement from an engineering perspective. +Some data are shared between Cells. +It creates hotspots/scalability issues in a system that might impact the experience of Organizations on other Cells, for instance during abuse. + +### 3.5. GitLab CI Daemon + +Another potential solution to explore is to have a dedicated service responsible for builds queueing, owning its database and working in a model of either sharded or Cell-ed service. +There were prior discussions about [CI/CD Daemon](https://gitlab.com/gitlab-org/gitlab/-/issues/19435). + +If the service is sharded: + +- Depending on the model, if runners are cluster-wide or Cell-local, this service would have to fetch data from all Cells. +- If the sharded service is used we could adapt a model of sharing a database containing `ci_pending_builds/ci_running_builds` with the service. +- If the sharded service is used we could consider a push model where each Cell pushes to CI/CD Daemon builds that should be picked by runner. +- The sharded service would be aware which Cell is responsible for processing the given build and could route processing requests to the designated Cell. + +If the service is Cell-ed: + +- All expectations of routable endpoints are still valid. + +In general usage of CI Daemon does not help significantly with the stated problem. +However, this offers a few upsides related to more efficient processing and decoupling model: push model and it opens a way to offer stateful communication with GitLab runners (ex. gRPC or Websockets). + +## 4. Evaluation + +Considering all options it appears that the most promising solution is to: + +- Use [Instance-wide are Cell-local](#33-instance-wide-are-cell-local) +- Refine endpoints to have routable identities (either via specific paths, or better tokens) + +Another potential upside is to get rid of `ci_builds.token` and rather use a `JWT token` that can much better and easier encode a wider set of scopes allowed by CI runner. + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/container-registry.md b/doc/architecture/blueprints/cells/impacted_features/container-registry.md new file mode 100644 index 00000000000..ea15dd52d94 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/container-registry.md @@ -0,0 +1,114 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Container Registry' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Container Registry + +GitLab [Container Registry](../../../../user/packages/container_registry/index.md) is a feature allowing to store Docker container images in GitLab. + +## 1. Definition + +GitLab Container Registry is a complex service requiring usage of PostgreSQL, Redis and Object Storage dependencies. +Right now there's undergoing work to introduce [Container Registry Metadata](../../container_registry_metadata_database/index.md) to optimize data storage and image retention policies of Container Registry. + +GitLab Container Registry is serving as a container for stored data, but on its own does not authenticate `docker login`. +The `docker login` is executed with user credentials (can be `personal access token`) or CI build credentials (ephemeral `ci_builds.token`). + +Container Registry uses data deduplication. +It means that the same blob (image layer) that is shared between many Projects is stored only once. +Each layer is hashed by `sha256`. + +The `docker login` does request a JWT time-limited authentication token that is signed by GitLab, but validated by Container Registry service. +The JWT token does store all authorized scopes (`container repository images`) and operation types (`push` or `pull`). +A single JWT authentication token can have many authorized scopes. +This allows Container Registry and client to mount existing blobs from other scopes. +GitLab responds only with authorized scopes. +Then it is up to GitLab Container Registry to validate if the given operation can be performed. + +The GitLab.com pages are always scoped to a Project. +Each Project can have many container registry images attached. + +Currently, on GitLab.com the actual registry service is served via `https://registry.gitlab.com`. + +The main identifiable problems are: + +- The authentication request (`https://gitlab.com/jwt/auth`) that is processed by GitLab.com. +- The `https://registry.gitlab.com` that is run by an external service and uses its own data store. +- Data deduplication. The Cells architecture with registry run in a Cell would reduce efficiency of data storage. + +## 2. Data flow + +### 2.1. Authorization request that is send by `docker login` + +```shell +curl \ + --user "username:password" \ + "https://gitlab/jwt/auth?client_id=docker&offline_token=true&service=container_registry&scope=repository:gitlab-org/gitlab-build-images:push,pull" +``` + +Result is encoded and signed JWT token. Second base64 encoded string (split by `.`) contains JSON with authorized scopes. + +```json +{"auth_type":"none","access":[{"type":"repository","name":"gitlab-org/gitlab-build-images","actions":["pull"]}],"jti":"61ca2459-091c-4496-a3cf-01bac51d4dc8","aud":"container_registry","iss":"omnibus-gitlab-issuer","iat":1669309469,"nbf":166} +``` + +### 2.2. Docker client fetching tags + +```shell +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/tags/list + +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/manifests/danger-ruby-2.6.6 +``` + +### 2.3. Docker client fetching blobs and manifests + +```shell +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/blobs/sha256:a3f2e1afa377d20897e08a85cae089393daa0ec019feab3851d592248674b416 +``` + +## 3. Proposal + +### 3.1. Shard Container Registry separately to Cells architecture + +Due to its extensive and in general highly scalable horizontal architecture it should be evaluated if the GitLab Container Registry should be run not in Cell, but in a Cluster and be scaled independently. +This might be easier, but would definitely not offer the same amount of data isolation. + +### 3.2. Run Container Registry within a Cell + +It appears that except `/jwt/auth` which would likely have to be processed by Router (to decode `scope`) the Container Registry could be run as a local service of a Cell. +The actual data at least in case of GitLab.com is not forwarded via registry, but rather served directly from Object Storage / CDN. + +Its design encodes container repository image in a URL that is easily routable. +It appears that we could re-use the same stateless Router service in front of Container Registry to serve manifests and blobs redirect. + +The only downside is increased complexity of managing standalone registry for each Cell, but this might be desired approach. + +## 4. Evaluation + +There do not seem to be any theoretical problems with running GitLab Container Registry in a Cell. +It seems that the service can be easily made routable to work well. +The practical complexities are around managing a complex service from an infrastructure side. + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/contributions-forks.md b/doc/architecture/blueprints/cells/impacted_features/contributions-forks.md new file mode 100644 index 00000000000..2053b87b125 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/contributions-forks.md @@ -0,0 +1,163 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Contributions: Forks' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the Cells design. +Significant aspects are not documented, though we expect to add them in the future. +This is one possible architecture for Cells, and we intend to contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that we can document the reasons for not choosing this approach. + +# Cells: Contributions: Forks + +The [forking workflow](../../../../user/project/repository/forking_workflow.md) allows users to copy existing Project sources into their own namespace of choice (personal or Group). + +## 1. Definition + +The [forking workflow](../../../../user/project/repository/forking_workflow.md) is a common workflow with various usage patterns: + +- It allows users to contribute back to an upstream Project. +- It persists repositories into their personal namespace. +- Users can copy a Project to make changes and release it as a modified Project. + +Forks allow users not having write access to a parent Project to make changes. +The forking workflow is especially important for the open-source community to contribute back to public Projects. +However, it is equally important in some companies that prefer a strong split of responsibilities and tighter access control. +The access to a Project is restricted to a designated list of developers. + +Forks enable: + +- Tighter control of who can modify the upstream Project. +- Split of responsibilities: Parent Project might use CI configuration connecting to production systems. +- To run CI pipelines in the context of a fork in a much more restrictive environment. +- To consider all forks to be unvetted which reduces risks of leaking secrets, or any other information tied to the Project. + +The forking model is problematic in a Cells architecture for the following reasons: + +- Forks are clones of existing repositories. Forks could be created across different Organizations, Cells and Gitaly shards. +- Users can create merge requests and contribute back to an upstream Project. This upstream Project might be in a different Organization and Cell. +- The merge request CI pipeline is executed in the context of the source Project, but presented in the context of the target Project. + +## 2. Data exploration + +From a [data exploration](https://gitlab.com/gitlab-data/product-analytics/-/issues/1380), we retrieved the following information about existing forks: + +- Roughly 1.8m forks exist on GitLab.com at the moment. +- The majority of forks are under a personal namespace (82%). +- We were expecting a minimal use of forks within the same top-level Group and/or organization. Forking is only necessary for users who don't have permissions to access a Project. Inside companies we wouldn't expect teams to use forking workflows much unless they for some reason have different permissions across different team members. The data showed that only 9% of fork relationships have matching ultimate parent namespace identifiers (top-level Groups and personal namespaces). The other 91% of fork relationships are forked across different top-level namespaces. When trying to match top-level Groups to an identifiable company, we saw that: + - 3% of forked Projects are forked from an upstream Project in the same organization. + - 83% of forked Projects do not have an identifiable organization related to either up or downstream Project. + - The remaining 14% are forked from a source Project within a different company. +- 9% of top-level Groups (95k) with activity in the last 12 months have a project with a fork relationship, compared to 5% of top-level Groups (91k) with no activity in the last 12 months. We expect these top-level Groups to be impacted by Cells. + +## 3. Proposal - Forks are created in a dedicated contribution space of the current Organization + +Instead of creating Projects across Organizations, forks are created in a contribution space tied to the Organization. +A contribution space is similar to a personal namespace but rather than existing in the default Organization, it exists within the Organization someone is trying to contribute to. +Example: + +- Any User that can view an Organization (all Users for public Organizations) can create a contribution space in the Organization. This is a dedicated namespace where they can create forks of Projects in that Organization. For example for `Produce Inc.` it could be `gitlab.com/organization/produce-inc/@ayufan`. +- To create a contribution space we do not require membership of an Organization as this would prevent open source workflows where contributors are able to fork and create a merge request without ever being invited to a Group or Project. We strictly respect visibility, so Users would not be able to create a fork in a private Organization without first being invited. +- When creating a fork for a Project Users will only be presented with the option to create forks in Groups that are part of the Organization. We will also give Users the option to create a contribution space and put the fork there. Today there is also a "Create a group" option when creating a fork. This functionality would also be limited to creating a new group in the organization to store the new fork. +- In order to support Users that want to fork without contributing back we might consider an option to create [an unlinked fork](../../../../user/project/repository/forking_workflow.md#unlink-a-fork) in any namespace they have permission to write to. +- The User has as many contribution spaces as Organizations they contribute to. +- The User cannot create additional personal Projects within contribution spaces. Personal Projects can continue to be created in their personal namespace. +- The Organization can prevent or disable usage of contribution spaces. This would disable forking by anyone that does not belong to a Group within the Organization. +- All current forks are migrated into the contribution space of the User in an Organization. Because this may result in data loss when the fork also has links to data outside of the upstream Project we will also keep the personal Project around as archived and remove the fork relationship. +- All forks are part of the Organization. +- Forks are not federated features. +- The contribution space and forked Project do not share configuration with the parent Project. +- If the Organization is deleted, the Projects containing forks will be moved either to the default Organization or we'll create a new Organization to house them, which is essentially a ghost Organization of the former Organization. +- Data in contribution spaces do not contribute to customer usage from a billing perspective. +- Today we do not have organization-scoped runners but if we do implement that they will likely need special settings for how or if they can be used by contribution space projects. + +## 4. Alternative proposals considered + +### 4.1. Intra-cluster forks + +This proposal implements forks as intra-Cluster forks where communication is done via API between all trusted Cells of a cluster: + +- Forks are created always in the context of a user's choice of Group. +- Forks are isolated from the Organization. +- Organization or Group owner could disable forking across Organizations, or forking in general. +- A merge request is created in the context of the target Project, referencing the external Project on another Cell. +- To target Project the merge reference is transferred that is used for presenting information in context of the target Project. +- CI pipeline is fetched in the context of the source Project as it is today, the result is fetched into the merge request of the target Project. +- The Cell holding the target Project internally uses GraphQL to fetch the status of the source Project and includes in context of the information for merge request. + +Pros: + +- All existing forks continue to work as they are, as they are treated as intra-Cluster forks. + +Cons: + +- The purpose of Organizations is to provide strong isolation between Organizations. Allowing to fork across does break security boundaries. +- However, this is no different to the ability of users today to clone a repository to a local computer and push it to any repository of choice. +- Access control of the source Project can be lower than that of the target Project. Today, the system requires that in order to contribute back, the access level needs to be the same for fork and upstream Project. + +### 4.2. Forks are created as internal Projects under current Projects + +Instead of creating Projects across Organizations, forks are attachments to existing Projects. +Each user forking a Project receives their unique Project. +Example: + +- For Project: `gitlab.com/gitlab-org/gitlab`, forks would be created in `gitlab.com/gitlab-org/gitlab/@kamil-gitlab`. +- Forks are created in the context of the current Organization, they do not cross Organization boundaries and are managed by the Organization. +- Tied to the user (or any other user-provided name of the fork). +- Forks are not federated features. + +Cons: + +- Does not answer how to handle and migrate all existing forks. +- Might share current Group/Project settings, which could be breaking some security boundaries. + +## 5. Evaluation + +### 5.1. Pros + +### 5.2. Cons + +## 6. Example + +As an example, we will demonstrate the impact of this proposal for the case that we move `gitlab-org/gitlab` to a different Organization. +`gitlab-org/gitlab` has [over 8K forks](https://gitlab.com/gitlab-org/gitlab/-/forks). + +### Does this direction impact the canonical URLs of those forks? + +Yes canonical URLs will change for forks. +Existing users that have forks in personal namespaces and want to continue contributing merge requests, will be required to migrate their fork to a new fork in a contribution space. +For example, a personal namespace fork at `https://gitlab.com/DylanGriffith/gitlab` will +need to be migrated to `https://gitlab.com/-/contributions/gitlab-inc/@DylanGriffith/gitlab`. +We may offer automated ways to move this, but manually the process would involve: + +1. Create the contribution space fork +1. Push your local branch from your original fork to the new fork +1. Recreate any merge request that was still open and you wanted to merge + +### Does it impact the Git URL of the repositories themselves? + +Yes. +In the above the example the Git URL would change from +`gitlab.com:DylanGriffith/gitlab.git` to `gitlab.com:/-/contributions/gitlab-inc/@DylanGriffith/gitlab.git`. + +### Would there be any user action required to accept their fork being moved within an Organization or towards a contribution space? + +If we offer an automated process we'd present this as an option for the user as they will become the new owner of the contribution space. + +### Can we make promises that we will not break the existing forks of public Projects hosted on GitLab.com? + +Existing fork projects will not be deleted but their fork relationship will be +removed when the source project is moved to another Organization. +The owner of the open source project will be made aware that they will disconnect their +forks when they move the project which will require them to close all existing +merge requests from those forks. +There will need to be some process for keeping the history from these merge requests while effectively losing the ability to +collaborate on them or merge them. + +In the case of `gitlab-org/gitlab` we will attempt to give as much notice of this process and make this process as transparent as possible. +When we make the decision to move this project to an Organization we will seek additional +feedback about what would be the minimum amount of automated migrations necessary to be acceptable here. +But the workflow for contributors will change after the move so this will be a punctuated event regardless. diff --git a/doc/architecture/blueprints/cells/impacted_features/data-migration.md b/doc/architecture/blueprints/cells/impacted_features/data-migration.md new file mode 100644 index 00000000000..9ff661ddf68 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/data-migration.md @@ -0,0 +1,99 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Data migration' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Data migration + +It is essential for a Cells architecture to provide a way to migrate data out of big Cells into smaller ones. +This document describes various approaches to provide this type of split. + +We also need to handle cases where data is already violating the expected isolation constraints of Cells, for example references cannot span multiple Organizations. +We know that existing features like linked issues allowed users to link issues across any Projects regardless of their hierarchy. +There are many similar features. +All of this data will need to be migrated in some way before it can be split across different Cells. +This may mean some data needs to be deleted, or the feature needs to be changed and modelled slightly differently before we can properly split or migrate Organizations between Cells. + +Having schema deviations across different Cells, which is a necessary consequence of different databases, will also impact our ability to migrate data between Cells. +Different schemas impact our ability to reliably replicate data across Cells and especially impact our ability to validate that the data is correctly replicated. +It might force us to only be able to move data between Cells when the schemas are all in sync (slowing down deployments and the rebalancing process) or possibly only migrate from newer to older schemas which would be complex. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +### 3.1. Split large Cells + +A single Cell can only be divided into many Cells. +This is based on the principle that it is easier to create an exact clone of an existing Cell in many replicas out of which some will be made authoritative once migrated. +Keeping those replicas up-to-date with Cell 0 is also much easier due to pre-existing replication solutions that can replicate the whole systems: Geo, PostgreSQL physical replication, etc. + +1. All data of an Organization needs to not be divided across many Cells. +1. Split should be doable online. +1. New Cells cannot contain pre-existing data. +1. N Cells contain exact replica of Cell 0. +1. The data of Cell 0 is live replicated to as many Cells it needs to be split. +1. Once consensus is achieved between Cell 0 and N-Cells, the Organizations to be migrated away are marked as read-only cluster-wide. +1. The `routes` is updated on for all Organizations to be split to indicate an authoritative Cell holding the most recent data, like `gitlab-org` on `cell-100`. +1. The data for `gitlab-org` on Cell 0, and on other non-authoritative N-Cells are dormant and will be removed in the future. +1. All accesses to `gitlab-org` on a given Cell are validated about `cell_id` of `routes` to ensure that given Cell is authoritative to handle the data. + +#### More challenges of this proposal + +1. There is no streaming replication capability for Elasticsearch, but you could + snapshot the whole Elasticsearch index and recreate, but this takes hours. + It could be handled by pausing Elasticsearch indexing on the initial Cell during + the migration as indexing downtime is not a big issue, but this still needs + to be coordinated with the migration process. +1. Syncing Redis, Gitaly, CI Postgres, Main Postgres, registry Postgres, other + new data stores snapshots in an online system would likely lead to gaps + without a long downtime. You need to choose a sync point and at the sync + point you need to stop writes to perform the migration. The more data stores + there are to migrate at the same time the longer the write downtime for the + failover. We would also need to find a reliable place in the application to + actually block updates to all these systems with a high degree of + confidence. In the past we've only been confident by shutting down all Rails + services because any Rails process could write directly to any of these at + any time due to async workloads or other surprising code paths. +1. How to efficiently delete all the orphaned data. Locating all `ci_builds` + associated with half the Organizations would be very expensive if we have to + do joins. We haven't yet determined if we'd want to store an `organization_id` + column on every table, but this is the kind of thing it would be helpful for. + +### 3.2. Migrate Organization from an existing Cell + +This is different to split, as we intend to perform logical and selective replication of data belonging to a single Organization. +Today this type of selective replication is only implemented by Gitaly where we can migrate Git repository from a single Gitaly node to another with minimal downtime. + +In this model we would require identifying all resources belonging to a given Organization: database rows, object storage files, Git repositories, etc. and selectively copy them over to another (likely) existing Cell importing data into it. +Ideally ensuring that we can perform logical replication live of all changed data, but change similarly to split which Cell is authoritative for this Organization. + +1. It is hard to identify all resources belonging to an Organization. +1. It requires either downtime for the Organization or a robust system to identify live changes made. +1. It likely will require a full database structure analysis (more robust than Project import/export) to perform selective PostgreSQL logical replication. + +#### More challenges of this proposal + +1. Logical replication is still not performant enough to keep up with our + scale. Even if we could use logical replication we still don't have an + efficient way to filter data related to a single Organization without + joining all the way to the `organizations` table which will slow down + logical replication dramatically. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/data-pipeline-ingestion.md b/doc/architecture/blueprints/cells/impacted_features/data-pipeline-ingestion.md new file mode 100644 index 00000000000..92de3df7c27 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/data-pipeline-ingestion.md @@ -0,0 +1,39 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Data Pipeline Ingestion' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Data pipeline ingestion + +The Cells architecture will have a significant impact on the current [data pipeline](https://about.gitlab.com/handbook/business-technology/data-team/platform/pipelines/SAAS-Gitlab-com/) which exports data from Postgres to Snowflake for the use of data analytics. This data pipeline fulfils many use cases (i.e. SAAS Service ping, Gainsight metrics and Reporting and Analytics of the SAAS Platform). + +## 1. Definition + +## 2. Data flow + +The current data pipeline is limited by not having the possibility to get data via a CDC mechanism (which leads to data quality issues) and works by polling the Postgres database and looking for new and updated records or fully extracting data for certain tables which causes a lot of overhead. +At the moment the data pipeline runs against two instances that get created from a snapshot of both the `main` and `ci` databases. +This is done to avoid workload on the production databases. +In the Cells architecture there will be more Postgres instances because of which the current pipeline couldn't scale to pull data from all the Postgres instances. Requirements around the data pipeline moving forward are as follows: + +- We need a process that allows capturing all the CDC (insert, update and delete) from all Cells, scaling automatically with N number of Cells. +- We need to have (direct or indirect) access to database instances which allows it to do data catch up in case of major failure or root cause analysis for data anomalies. +- We need monitoring in place to alert any incident that can delay the data ingestion. + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/database-sequences.md b/doc/architecture/blueprints/cells/impacted_features/database-sequences.md new file mode 100644 index 00000000000..2aeaaed7d64 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/database-sequences.md @@ -0,0 +1,74 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Database Sequences' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Database Sequences + +GitLab today ensures that every database row create has a unique ID, allowing to access a merge request, CI Job or Project by a known global ID. +Cells will use many distinct and not connected databases, each of them having a separate ID for most entities. +At a minimum, any ID referenced between a Cell and the shared schema will need to be unique across the cluster to avoid ambiguous references. +Further to required global IDs, it might also be desirable to retain globally unique IDs for all database rows to allow migrating resources between Cells in the future. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +These are some preliminary ideas how we can retain unique IDs across the system. + +### 3.1. UUID + +Instead of using incremental sequences, use UUID (128 bit) that is stored in the database. + +- This might break existing IDs and requires adding a UUID column for all existing tables. +- This makes all indexes larger as it requires storing 128 bit instead of 32/64 bit in index. + +### 3.2. Use Cell index encoded in ID + +Because a significant number of tables already use 64 bit ID numbers we could use MSB to encode the Cell ID: + +- This might limit the amount of Cells that can be enabled in a system, as we might decide to only allocate 1024 possible Cell numbers. +- This would make it possible to migrate IDs between Cells, because even if an entity from Cell 1 is migrated to Cell 100 this ID would still be unique. +- If resources are migrated the ID itself will not be enough to decode the Cell number and we would need a lookup table. +- This requires updating all IDs to 32 bits. + +### 3.3. Allocate sequence ranges from central place + +Each Cell might receive its own range of sequences as they are consumed from a centrally managed place. +Once a Cell consumes all IDs assigned for a given table it would be replenished and a next range would be allocated. +Ranges would be tracked to provide a faster lookup table if a random access pattern is required. + +- This might make IDs migratable between Cells, because even if an entity from Cell 1 is migrated to Cell 100 this ID would still be unique. +- If resources are migrated the ID itself will not be enough to decode the Cell number and we would need a much more robust lookup table as we could be breaking previously assigned sequence ranges. +- This does not require updating all IDs to 64 bits. +- This adds some performance penalty to all `INSERT` statements in Postgres or at least from Rails as we need to check for the sequence number and potentially wait for our range to be refreshed from the ID server. +- The available range will need to be stored and incremented in a centralized place so that concurrent transactions cannot possibly get the same value. + +### 3.4. Define only some tables to require unique IDs + +Maybe it is acceptable only for some tables to have a globally unique IDs. It could be Projects, Groups and other top-level entities. +All other tables like `merge_requests` would only offer a Cell-local ID, but when referenced outside it would rather use an IID (an ID that is monotonic in context of a given resource, like a Project). + +- This makes the ID 10000 for `merge_requests` be present on all Cells, which might be sometimes confusing regarding the uniqueness of the resource. +- This might make random access by ID (if ever needed) impossible without using a composite key, like: `project_id+merge_request_id`. +- This would require us to implement a transformation/generation of new ID if we need to migrate records to another Cell. This can lead to very difficult migration processes when these IDs are also used as foreign keys for other records being migrated. +- If IDs need to change when moving between Cells this means that any links to records by ID would no longer work even if those links included the `project_id`. +- If we plan to allow these IDs to not be unique and change the unique constraint to be based on a composite key then we'd need to update all foreign key references to be based on the composite key. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/explore.md b/doc/architecture/blueprints/cells/impacted_features/explore.md new file mode 100644 index 00000000000..f839ce4be34 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/explore.md @@ -0,0 +1,71 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Explore' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Explore + +Explore may not play a critical role in GitLab as it functions today, but GitLab today is not isolated. It is the isolation that makes Explore or some viable replacement necessary. + +The existing Group and Project Explore will initially be scoped to an Organization. However, there is a need for a global Explore that spans across Organizations to support the discoverability of public Groups and Projects, in particular in the context of discovering open source Projects. See user feedback [here](https://gitlab.com/gitlab-org/gitlab/-/issues/21582#note_1458298192) and [here](https://gitlab.com/gitlab-org/gitlab/-/issues/418228#note_1470045468). + +## 1. Definition + +The Explore functionality helps users in discovering Groups and Projects. Unauthenticated Users are only able to explore public Groups and Projects, authenticated Users can see all the Groups and Projects that they have access to, including private and internal Groups and Projects. + +## 2. Data flow + +## 3. Proposal + +The Explore feature problem falls under the broader umbrella of solving inter-Cell communication. [This topic warrants deeper research](../index.md#can-different-cells-communicate-with-each-other). + +Below are possible directions for further investigation. + +### 3.1. Read only table mirror + +- Create a `shared_projects` table in the shared cluster-wide database. +- The model for this table is read-only. No inserts/updates/deletes are allowed. +- The table is filled with data (or a subset of data) from the Projects Cell-local table. + - The write model Project (which is Cell-local) writes to the local database. We will primarily use this model for anything Cell-local. + - This data is synchronized with `shared_projects` via a background job any time something changes. + - The data in `shared_projects` is stored normalized, so that all the information necessary to display the Project Explore is there. +- The Project Explore (as of today) is part of an instance-wide functionality, since it's not namespaced to any organizations/groups. + - This section will read data using the read model for `shared_projects`. +- Once the user clicks on a Project, they are redirected to the Cell containing the Organization. + +Downsides: + +- Need to have an explicit pattern to access instance-wide data. This however may be useful for admin functionalities too. +- The Project Explore may not be as rich in features as it is today (various filtering options, role you have on that Project, etc.). +- Extra complexity in managing CQRS. + +### 3.2 Explore scoped to an Organization + +The Project Explore and Group Explore are scoped to an Organization. + +Downsides: + +- No global discoverability of Groups and Projects. + +## 4. Evaluation + +The existing Group and Project Explore will initially be scoped to an Organization. Considering the [current usage of the Explore feature](https://gitlab.com/gitlab-data/product-analytics/-/issues/1302#note_1491215521), we deem this acceptable. Since all existing Users, Groups and Projects will initially be part of the default Organization, Groups and Projects will remain explorable and accessible as they are today. Only once existing Groups and Projects are moved out of the default Organization into different Organizations will this become a noticeable problem. Solutions to mitigate this are discussed in [issue #418228](https://gitlab.com/gitlab-org/gitlab/-/issues/418228). Ultimately, Explore could be replaced with a better search experience altogether. + +## 4.1. Pros + +- Initially the lack of discoverability will not be a problem. +- Only around [1.5% of all exisiting Users are using the Explore functionality on a monthly basis](https://gitlab.com/gitlab-data/product-analytics/-/issues/1302#note_1491215521). + +## 4.2. Cons + +- The GitLab owned top-level Groups would be some of the first to be moved into their own Organization and thus be detached from the explorability of the default Organization. diff --git a/doc/architecture/blueprints/cells/impacted_features/git-access.md b/doc/architecture/blueprints/cells/impacted_features/git-access.md new file mode 100644 index 00000000000..611b4db5f43 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/git-access.md @@ -0,0 +1,156 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Git Access' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Git Access + +This document describes impact of Cells architecture on all Git access (over HTTPS and SSH) patterns providing explanation of how potentially those features should be changed to work well with Cells. + +## 1. Definition + +Git access is done throughout the application. +It can be an operation performed by the system (read Git repository) or by a user (create a new file via Web IDE, `git clone` or `git push` via command line). +The Cells architecture defines that all Git repositories will be local to the Cell, so no repository could be shared with another Cell. + +The Cells architecture will require that any Git operation can only be handled by a Cell holding the data. +It means that any operation either via Web interface, API, or GraphQL needs to be routed to the correct Cell. +It means that any `git clone` or `git push` operation can only be performed in the context of a Cell. + +## 2. Data flow + +The are various operations performed today by GitLab on a Git repository. +This describes the data flow how they behave today to better represent the impact. + +It appears that Git access does require changes only to a few endpoints that are scoped to a Project. +There appear to be different types of repositories: + +- Project: assigned to Group +- Wiki: additional repository assigned to Project +- Design: similar to Wiki, additional repository assigned to Project +- Snippet: creates a virtual Project to hold repository, likely tied to the User + +### 2.1. Git clone over HTTPS + +Execution of: `git clone` over HTTPS + +```mermaid +sequenceDiagram + User ->> Workhorse: GET /gitlab-org/gitlab.git/info/refs?service=git-upload-pack + Workhorse ->> Rails: GET /gitlab-org/gitlab.git/info/refs?service=git-upload-pack + Rails ->> Workhorse: 200 OK + Workhorse ->> Gitaly: RPC InfoRefsUploadPack + Gitaly ->> User: Response + User ->> Workhorse: POST /gitlab-org/gitlab.git/git-upload-pack + Workhorse ->> Gitaly: RPC PostUploadPackWithSidechannel + Gitaly ->> User: Response +``` + +### 2.2. Git clone over SSH + +Execution of: `git clone` over SSH + +```mermaid +sequenceDiagram + User ->> Git SSHD: ssh git@gitlab.com + Git SSHD ->> Rails: GET /api/v4/internal/authorized_keys + Rails ->> Git SSHD: 200 OK (list of accepted SSH keys) + Git SSHD ->> User: Accept SSH + User ->> Git SSHD: git clone over SSH + Git SSHD ->> Rails: POST /api/v4/internal/allowed?project=/gitlab-org/gitlab.git&service=git-upload-pack + Rails ->> Git SSHD: 200 OK + Git SSHD ->> Gitaly: RPC SSHUploadPackWithSidechannel + Gitaly ->> User: Response +``` + +### 2.3. Git push over HTTPS + +Execution of: `git push` over HTTPS + +```mermaid +sequenceDiagram + User ->> Workhorse: GET /gitlab-org/gitlab.git/info/refs?service=git-receive-pack + Workhorse ->> Rails: GET /gitlab-org/gitlab.git/info/refs?service=git-receive-pack + Rails ->> Workhorse: 200 OK + Workhorse ->> Gitaly: RPC PostReceivePack + Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111&service=git-receive-pack + Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 + Gitaly ->> User: Response +``` + +### 2.4. Git push over SSHD + +Execution of: `git clone` over SSH + +```mermaid +sequenceDiagram + User ->> Git SSHD: ssh git@gitlab.com + Git SSHD ->> Rails: GET /api/v4/internal/authorized_keys + Rails ->> Git SSHD: 200 OK (list of accepted SSH keys) + Git SSHD ->> User: Accept SSH + User ->> Git SSHD: git clone over SSH + Git SSHD ->> Rails: POST /api/v4/internal/allowed?project=/gitlab-org/gitlab.git&service=git-receive-pack + Rails ->> Git SSHD: 200 OK + Git SSHD ->> Gitaly: RPC ReceivePack + Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 + Gitaly ->> User: Response +``` + +### 2.5. Create commit via Web + +Execution of `Add CHANGELOG` to repository: + +```mermaid +sequenceDiagram + Web ->> Puma: POST /gitlab-org/gitlab/-/create/main + Puma ->> Gitaly: RPC TreeEntry + Gitaly ->> Rails: POST /api/v4/internal/allowed?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/pre_receive?gl_repository=project-111 + Gitaly ->> Rails: POST /api/v4/internal/post_receive?gl_repository=project-111 + Gitaly ->> Puma: Response + Puma ->> Web: See CHANGELOG +``` + +## 3. Proposal + +The Cells stateless router proposal requires that any ambiguous path (that is not routable) will be made routable. +It means that at least the following paths will have to be updated to introduce a routable entity (Project, Group, or Organization). + +Change: + +- `/api/v4/internal/allowed` => `/api/v4/internal/projects/<gl_repository>/allowed` +- `/api/v4/internal/pre_receive` => `/api/v4/internal/projects/<gl_repository>/pre_receive` +- `/api/v4/internal/post_receive` => `/api/v4/internal/projects/<gl_repository>/post_receive` +- `/api/v4/internal/lfs_authenticate` => `/api/v4/internal/projects/<gl_repository>/lfs_authenticate` + +Where: + +- `gl_repository` can be `project-1111` (`Gitlab::GlRepository`) +- `gl_repository` in some cases might be a full path to repository as executed by GitLab Shell (`/gitlab-org/gitlab.git`) + +## 4. Evaluation + +Supporting Git repositories if a Cell can access only its own repositories does not appear to be complex. +The one major complication is supporting snippets, but this likely falls in the same category as for the approach to support a user's Personal Namespace. + +## 4.1. Pros + +1. The API used for supporting HTTPS/SSH and Hooks are well defined and can easily be made routable. + +## 4.2. Cons + +1. The sharing of repositories objects is limited to the given Cell and Gitaly node. +1. Cross-Cells forks are likely impossible to be supported (discover: How this works today across different Gitaly node). diff --git a/doc/architecture/blueprints/cells/impacted_features/gitlab-pages.md b/doc/architecture/blueprints/cells/impacted_features/gitlab-pages.md new file mode 100644 index 00000000000..7e4ab785095 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/gitlab-pages.md @@ -0,0 +1,30 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: GitLab Pages' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: GitLab Pages + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/global-search.md b/doc/architecture/blueprints/cells/impacted_features/global-search.md new file mode 100644 index 00000000000..475db381ff5 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/global-search.md @@ -0,0 +1,35 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Global search' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Global search + +When we introduce multiple Cells we intend to isolate all services related to those Cells. +This will include Elasticsearch which means our current global search functionality will not work. +It may be possible to implement aggregated search across all Cells, but it is unlikely to be performant to do fan-out searches across all Cells especially once you start to do pagination which requires setting the correct offset and page number for each search. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +Likely the first versions of Cells will not support global searches. +Later, we may consider if building global searches to support popular use cases is worthwhile. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/graphql.md b/doc/architecture/blueprints/cells/impacted_features/graphql.md new file mode 100644 index 00000000000..e8850dfbee3 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/graphql.md @@ -0,0 +1,81 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: GraphQL' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: GraphQL + +GitLab extensively uses GraphQL to perform efficient data query operations. +GraphQL due to it's nature is not directly routable. +The way GitLab uses it calls the `/api/graphql` endpoint, and only the query or mutation of the body request might define where the data can be accessed. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +There are at least two main ways to implement GraphQL in a Cells architecture. + +### 3.1. GraphQL routable by endpoint + +Change `/api/graphql` to `/api/organization/<organization>/graphql`. + +- This breaks all existing usages of `/api/graphql` endpoint because the API URI is changed. + +### 3.2. GraphQL routable by body + +As part of router parse GraphQL body to find a routable entity, like `project`. + +- This still makes the GraphQL query be executed only in context of a given Cell and not allowing the data to be merged. + +```json +# Good example +{ + project(fullPath:"gitlab-org/gitlab") { + id + description + } +} + +# Bad example, since Merge Request is not routable +{ + mergeRequest(id: 1111) { + iid + description + } +} +``` + +### 3.3. Merging GraphQL Proxy + +Implement as part of router GraphQL Proxy which can parse body and merge results from many Cells. + +- This might make pagination hard to achieve, or we might assume that we execute many queries of which results are merged across all Cells. + +```json +{ + project(fullPath:"gitlab-org/gitlab"){ + id, description + } + group(fullPath:"gitlab-com") { + id, description + } +} +``` + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/organizations.md b/doc/architecture/blueprints/cells/impacted_features/organizations.md new file mode 100644 index 00000000000..9fc8241e852 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/organizations.md @@ -0,0 +1,36 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Organizations' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Organizations + +One of the major designs of a Cells architecture is strong isolation between Groups. +Organizations as described by the [Organization blueprint](../../organization/index.md) provides a way to have plausible UX for joining together many Groups that are isolated from the rest of the system. + +## 1. Definition + +Cells do require that all Groups and Projects of a single Organization can only be stored on a single Cell because a Cell can only access data that it holds locally and has very limited capabilities to read information from other Cells. + +Cells with Organizations do require strong isolation between Organizations. + +It will have significant implications on various user-facing features, like Todos, dropdowns allowing to select Projects, references to other issues or Projects, or any other social functions present at GitLab. +Today those functions were able to reference anything in the whole system. +With the introduction of Organizations this will be forbidden. + +This problem definition aims to answer effort and implications required to add strong isolation between Organizations to the system, including features affected and their data processing flow. +The purpose is to ensure that our solution when implemented consistently avoids data leakage between Organizations residing on a single Cell. + +## 2. Proposal + +See the [Organization blueprint](../../organization/index.md). diff --git a/doc/architecture/blueprints/cells/impacted_features/personal-access-tokens.md b/doc/architecture/blueprints/cells/impacted_features/personal-access-tokens.md new file mode 100644 index 00000000000..3aca9f1e116 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/personal-access-tokens.md @@ -0,0 +1,31 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Personal Access Tokens' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Personal Access Tokens + +## 1. Definition + +Personal Access Tokens associated with a User are a way for Users to interact with the API of GitLab to perform operations. +Personal Access Tokens today are scoped to the User, and can access all Groups that a User has access to. + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/personal-namespaces.md b/doc/architecture/blueprints/cells/impacted_features/personal-namespaces.md new file mode 100644 index 00000000000..55d974bb351 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/personal-namespaces.md @@ -0,0 +1,71 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Personal Namespaces' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the Cells design. +Significant aspects are not documented, though we expect to add them in the future. +This is one possible architecture for Cells, and we intend to contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that we can document the reasons for not choosing this approach. + +# Cells: Personal Namespaces + +Personal Namespaces do not easily fit with our overall architecture in Cells because the Cells architecture depends on all data belonging to a single Organization. +When Users are allowed to work across multiple Organizations there is no natural fit for picking a single Organization to store personal Namespaces and their Projects. + +One important engineering constraint in Cells will be that data belonging to some Organization should not be linked to data belonging to another Organization. +And specifically that functionality in GitLab can be scoped to a single Organization at a time. +This presents a challenge for personal Namespaces as forking is one of the important workloads for personal Namespaces. +Functionality related to forking and the UI that presents forked MRs to users will often require data from both the downstream and upstream Projects at the same time. +Implementing such functionality would be very difficult if that data belonged in different Organizations stored on different +Cells. +This is especially the case with the merge request, as it is one of the most complicated and performance critical features in GitLab. + +Today personal Namespaces serve two purposes that are mostly non-overlapping: + +1. They provide a place for users to create personal Projects + that aren't expected to receive contributions from other people. This use case saves them from having to create a Group just for themselves. +1. They provide a default place for a user to put any forks they + create when contributing to Projects where they don't have permission to push a branch. This again saves them from needing to create a Group just to store these forks. But the primary user need here is because they can't push branches to the upstream Project so they create a fork and contribute merge requests from the fork. + +## 1. Definition + +A [personal Namespace](../../../../user/namespace/index.md#types-of-namespaces) is based on a username and provided when a user creates an account. +Users can create [personal Projects](../../../../user/project/working_with_projects.md#view-personal-projects) under their personal Namespace. + +## 2. Data flow + +## 3. Proposal + +As described above, personal Namespaces serve two purposes today: + +1. A place for users to store their Projects to save them from creating a Group. +1. A place for users to store forks when they want to contribute to a Project where they don't have permission to push a branch. + +In this proposal we will only focus on (1) and assume that (2) will be replaced by suitable workflows described in [Cells: Contributions: Forks](../impacted_features/contributions-forks.md). + +Since we plan to move away from using personal Namespaces as a home for storing forks, we can assume that the main remaining use case does not need to support cross-Organization linking. +In this case the easiest thing to do is to keep all personal Namespaces in the default Organization. +Depending on the amount of workloads happening in personal Namespaces we may be required in the future to migrate them to different Cells. +This may necessitate that they all get moved to some Organization created just for the user. +If we go this route, there may be breakage similar to what will happen to when we move Groups or Projects into their own Organization, though the full impact may need further investigation. + +This decision, however, means that existing personal Namespaces that were used as forks to contribute to some upstream Project will become disconnected from the upstream as soon as the upstream moves into an Organization. +On GitLab.com 10% of all projects in personal Namespaces are forks. +This may be a slightly disruptive workflow but as long as the forks are mainly just storing branches used in merge requests then it may be reasonable to ask the affected users to recreate the fork in the context of the Organization. + +For existing Users, we suggest to keep their existing personal Namespaces in the default Organization. +New Users joining an Organization other than the default Organization will also have their personal Namespace hosted on the default Organization. Having all personal Namespaces in the default Organization means we don't need to worry about deletion of the parent organization and the impact of that on personal Namespaces, which would be the case if they existed in other organizations. +This implies that all Users will have an association to the default Organization via their personal Namespace, requiring them to switch to the default Organization to access their personal Namespace. + +We will further explore the idea of a `contribution space` to give Users a place to store forks when they want to contribute to a Project where they don't have permission to push a branch. +That discussion will be handled as part of the larger discussion of the [Cells impact on forks](../impacted_features/contributions-forks.md). + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/router-endpoints-classification.md b/doc/architecture/blueprints/cells/impacted_features/router-endpoints-classification.md new file mode 100644 index 00000000000..d403d6ff963 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/router-endpoints-classification.md @@ -0,0 +1,34 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Router Endpoints Classification' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Router Endpoints Classification + +Classification of all endpoints is essential to properly route requests hitting the load balancer of a GitLab installation to a Cell that can serve it. +Each Cell should be able to decode each request and classify which Cell it belongs to. + +GitLab currently implements hundreds of endpoints. +This document tries to describe various techniques that can be implemented to allow the Rails to provide this information efficiently. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/schema-changes.md b/doc/architecture/blueprints/cells/impacted_features/schema-changes.md new file mode 100644 index 00000000000..0d6ca4717d3 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/schema-changes.md @@ -0,0 +1,38 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Schema changes' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Schema changes + +When we introduce multiple Cells that own their own databases this will complicate the process of making schema changes to Postgres and Elasticsearch. +Today we already need to be careful to make changes comply with our zero downtime deployments. +For example, [when removing a column we need to make changes over 3 separate deployments](../../../../development/database/avoiding_downtime_in_migrations.md#dropping-columns). +We have tooling like `post_migrate` that helps with these kinds of changes to reduce the number of merge requests needed, but these will be complicated when we are dealing with deploying multiple Rails applications that will be at different versions at any one time. +This problem will be particularly tricky to solve for shared databases like our plan to share the `users` related tables among all Cells. + +A key benefit of Cells may be that it allows us to run different customers on different versions of GitLab. +We may choose to update our own Cell before all our customers giving us even more flexibility than our current canary architecture. +But doing this means that schema changes need to have even more versions of backward compatibility support which could slow down development as we need extra steps to make schema changes. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/secrets.md b/doc/architecture/blueprints/cells/impacted_features/secrets.md new file mode 100644 index 00000000000..681c229711d --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/secrets.md @@ -0,0 +1,43 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Secrets' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Secrets + +Where possible, each Cell should have its own distinct set of secrets. +However, there will be some secrets that will be required to be the same for all Cells in the cluster. + +## 1. Definition + +GitLab has a lot of [secrets](https://docs.gitlab.com/charts/installation/secrets.html) that need to be configured. +Some secrets are for inter-component communication, for example, `GitLab Shell secret`, and used only within a Cell. +Some secrets are used for features, for example, `ci_jwt_signing_key`. + +## 2. Data flow + +## 3. Proposal + +1. Secrets used for features will need to be consistent across all Cells, so that the UX is consistent. + 1. This is especially true for the `db_key_base` secret which is used for + encrypting data at rest in the database - so that Projects that are + transferred to another Cell will continue to work. We do not want to have + to re-encrypt such rows when we move Projects/Groups between Cells. +1. Secrets which are used for intra-Cell communication only should be uniquely generated + per Cell. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/snippets.md b/doc/architecture/blueprints/cells/impacted_features/snippets.md new file mode 100644 index 00000000000..f77879907df --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/snippets.md @@ -0,0 +1,56 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Snippets' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Snippets + +Snippets will be scoped to an Organization. Initially it will not be possible to aggregate snippet collections across Organizations. See also [issue #416954](https://gitlab.com/gitlab-org/gitlab/-/issues/416954). + +## 1. Definition + +Two different types of snippets exist: + +- [Project snippets](../../../../api/project_snippets.md). These snippets have URLs + like `/<group>/<project>/-/snippets/123` +- [Personal snippets](../../../../user/snippets.md). These snippets have URLs like + `/-/snippets/123` + +Snippets are backed by a Git repository. + +## 2. Data flow + +## 3. Proposal + +### 3.1. Scoped to an organization + +Both project and personal snippets will be scoped to an Organization. + +- Project snippets URLs will remain unchanged, as the URLs are routable. +- Personal snippets URLs will need to change to be `/-/organizations/<organization>/snippets/123`, + so that the URL is routeable + +Creation of snippets will also be scoped to a User's current Organization. Because of that, we recommend renaming `personal snippets` to `organization snippets` once the Organization is rolled out. A User can create many independent snippet collections across multiple Organizations. + +## 4. Evaluation + +Snippets are scoped to an Organization because Gitaly is confined to a Cell. + +## 4.1. Pros + +- No need to have clusterwide Gitaly. + +## 4.2. Cons + +- We will break [snippet discovery](/ee/user/snippets.md#discover-snippets). +- Snippet access may become subordinate to the visibility of the Organization. diff --git a/doc/architecture/blueprints/cells/impacted_features/template.md b/doc/architecture/blueprints/cells/impacted_features/template.md new file mode 100644 index 00000000000..3cece3dc99e --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/template.md @@ -0,0 +1,30 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Problem A' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: A + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/uploads.md b/doc/architecture/blueprints/cells/impacted_features/uploads.md new file mode 100644 index 00000000000..fdac3a9977c --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/uploads.md @@ -0,0 +1,30 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Uploads' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Uploads + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/cells/impacted_features/user-profile.md b/doc/architecture/blueprints/cells/impacted_features/user-profile.md new file mode 100644 index 00000000000..fac8552e6d6 --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/user-profile.md @@ -0,0 +1,53 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: User Profile' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: User Profile + +The existing User Profiles will initially be scoped to an Organization. Long-term, we should consider aggregating parts of the User activity across Organizations to enable Users a global view of their contributions. + +## 1. Definition + +Each GitLab account has a [User Profile](../../../../user/profile/index.md), which contains information about the User and their GitLab activity. + +## 2. Data flow + +## 3. Proposal + +User Profiles will be scoped to an Organization. We follow the same pattern as is used for `Your Work`, meaning that profiles are always seen in the context of an Organization. + +- User Profile URLs will reference the Organization with the following URL structure `/-/organizations/<organization>/username`. +- Users can set a Home Organization as their main Organization. +- The default User Profile URL `/<username>` will refer to the user's Home Organization, or the default Organization if the user's Home Organization is not set. +- Users who do not exist in the database at all display a 404 not found error when trying to access their User Profile. +- User who haven't contributed to an Organization display their User Profile with an empty state. +- When displaying a User Profile empty state, if the profile has a Home Organization set to another Organization, we display a call-to-action allowing navigation to the main Organization. +- Breadcrumbs on the User Profile will present as `[Organization Name] / [Username]`. + +See [issue #411931](https://gitlab.com/gitlab-org/gitlab/-/issues/411931) for design proposals. + +## 4. Evaluation + +We expect the [majority of Users to perform most of their activity in one single Organization](../../organization/index.md#data-exploration). +This is why we deem it acceptable to scope the User Profile to an Organization at first. +More discovery is necessary to understand which aspects of the current User Profile are relevant to showcase contributions in a global context. + +## 4.1. Pros + +- Viewing a User Profile scoped to an Organization allows you to focus on contributions that are most relevant to your Organization, filtering out the User's other activities. +- Existing User Profile URLs do not break. + +## 4.2. Cons + +- Users will lose the ability to display their entire activity, which may lessen the effectiveness of using their User Profile as a resume of achievements when working across multiple Organizations. diff --git a/doc/architecture/blueprints/cells/impacted_features/your-work.md b/doc/architecture/blueprints/cells/impacted_features/your-work.md new file mode 100644 index 00000000000..5a72d50fbea --- /dev/null +++ b/doc/architecture/blueprints/cells/impacted_features/your-work.md @@ -0,0 +1,60 @@ +--- +stage: enablement +group: Tenant Scale +description: 'Cells: Your Work' +--- + +<!-- vale gitlab.FutureTense = NO --> + +This document is a work-in-progress and represents a very early state of the +Cells design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Cells, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Cells: Your Work + +Your Work will be scoped to an Organization. +Counts presented in the individual dashboards will relate to the selected Organization. + +## 1. Definition + +When accessing `gitlab.com/dashboard/`, users can find a [focused view of items that they have access to](../../../../tutorials/left_sidebar/index.md#use-a-more-focused-view). +This overview contains dashboards relating to: + +- Projects +- Groups +- Issues +- Merge requests +- To-Do list +- Milestones +- Snippets +- Activity +- Workspaces +- Environments +- Operations +- Security + +## 2. Data flow + +## 3. Proposal + +Your Work will be scoped to an Organization, giving the user an overview of all the items they can access in the Organization they are currently viewing. + +- Issue, Merge request and To-Do list counts will refer to the selected Organization. +- The URL will reference the Organization with the following URL structure `/-/organizations/<organization>/dashboard`. +- The default URL `/dashboard` will refer to the [Home Organization](../impacted_features/user-profile.md#3-proposal). + +## 4. Evaluation + +Scoping Your Work to an Organization makes sense in the context of the [proposed Organization navigation](https://gitlab.com/gitlab-org/gitlab/-/issues/417778). +Considering that [we expect most users to work in a single Organization](../../organization/index.md#data-exploration), we deem this impact acceptable. + +## 4.1. Pros + +- Viewing Your Work scoped to an Organization allows Users to focus on content that is most relevant to their currently selected Organization. + +## 4.2. Cons + +- Users working across multiple Organizations will have to navigate to each Organization to access all of their work items. diff --git a/doc/architecture/blueprints/cells/index.md b/doc/architecture/blueprints/cells/index.md index 0e93b9d5d3b..28414f9b68c 100644 --- a/doc/architecture/blueprints/cells/index.md +++ b/doc/architecture/blueprints/cells/index.md @@ -1,5 +1,5 @@ --- -status: accepted +status: ongoing creation-date: "2022-09-07" authors: [ "@ayufan", "@fzimmer", "@DylanGriffith", "@lohrc", "@tkuah" ] coach: "@ayufan" @@ -18,9 +18,7 @@ Cells is a new architecture for our software as a service platform. This archite For more information about Cells, see also: -- [Glossary](glossary.md) -- [Goals](goals.md) -- [Cross-section impact](impact.md) +- [Goals, Glossary and Requirements](goals.md) ## Work streams @@ -101,6 +99,10 @@ The first 2-3 quarters are required to define a general split of data and build The purpose is to perform a targeted decomposition of `users` and `projects`, because `projects` will be stored locally in the Cell. +1. **User can create Organization on Cell 2.** + + The purpose is to create Organizations that are isolated from each other. + 1. **User can change profile avatar that is shared in cluster.** The purpose is to fix global uploads that are shared in cluster. @@ -166,6 +168,11 @@ For example: The routing service needs to be able to discover and monitor the health of all Cells. +1. **User can use single domain to interact with many Cells.** + + The routing service will intelligently route all requests to Cells based on the resource being + accessed versus the Cell containing the data. + 1. **Router endpoints classification.** The stateless routing service will fetch and cache information about endpoints from one of the Cells. @@ -258,28 +265,30 @@ One iteration describes one quarter's worth of work. - Data access layer: Initial Admin Area settings are shared across cluster. - Essential workflows: Allow to share cluster-wide data with database-level data access layer -1. [Iteration 2](https://gitlab.com/groups/gitlab-org/-/epics/9813) - FY24Q2 - In progress +1. [Iteration 2](https://gitlab.com/groups/gitlab-org/-/epics/9813) - Expected delivery: 16.2 FY24Q2 | Actual delivery: 16.4 FY24Q3 - In progress - Essential workflows: User accounts are shared across cluster. - Essential workflows: User can create Group. -1. [Iteration 3](https://gitlab.com/groups/gitlab-org/-/epics/10997) - FY24Q3 - Planned +1. [Iteration 3](https://gitlab.com/groups/gitlab-org/-/epics/10997) - Expected delivery: 16.7 FY24Q4 - Planned - Essential workflows: User can create Project. - Routing: Technology. - - Data access layer: Evaluate the efficiency of database-level access vs. API-oriented access layer + - Routing: Cell discovery. + - Data access layer: Evaluate the efficiency of database-level access vs. API-oriented access layer. + - Data access layer: Data access layer. -1. [Iteration 4](https://gitlab.com/groups/gitlab-org/-/epics/10998) - FY24Q4 +1. [Iteration 4](https://gitlab.com/groups/gitlab-org/-/epics/10998) - Expected delivery: 16.10 FY25Q1 - Planned - - Essential workflows: User can push to Git repository. - - Essential workflows: User can create issue, merge request, and merge it after it is green. + - Essential workflows: User can create organization on Cell 2. - Data access layer: Cluster-unique identifiers. - - Routing: Cell discovery. - - Routing: Router endpoints classification. - - Cell deployment: Extend GitLab Dedicated to support GCP + - Routing: User can use single domain to interact with many Cells. + - Cell deployment: Extend GitLab Dedicated to support GCP. -1. Iteration 5 - FY25Q1 +1. Iteration 5..N - starting FY25Q1 + - Essential workflows: User can push to Git repository. + - Essential workflows: User can create issue, merge request, and merge it after it is green. - Essential workflows: User can run CI pipeline. - Essential workflows: Instance-wide settings are shared across cluster. - Essential workflows: User can change profile avatar that is shared in cluster. @@ -287,22 +296,13 @@ One iteration describes one quarter's worth of work. - Essential workflows: User can manage Group and Project members. - Essential workflows: User can manage instance-wide runners. - Essential workflows: User is part of Organization and can only see information from the Organization. + - Routing: Router endpoints classification. - Routing: GraphQL and other ambiguous endpoints. - Data access layer: Allow to share cluster-wide data with database-level data access layer. - Data access layer: Cluster-wide deletions. - - Data access layer: Data access layer. - Data access layer: Database migrations. -1. Iteration 6 - FY25Q2 - - TBD - -1. Iteration 7 - FY25Q3 - - TBD - -1. Iteration 8 - FY25Q4 - - TBD - -## Technical Proposals +## Technical proposals The Cells architecture has long lasting implications to data processing, location, scalability and the GitLab architecture. This section links all different technical proposals that are being evaluated. @@ -315,34 +315,36 @@ This section links all different technical proposals that are being evaluated. The Cells architecture will impact many features requiring some of them to be rewritten, or changed significantly. Below is a list of known affected features with preliminary proposed solutions. -- [Cells: Admin Area](cells-feature-admin-area.md) -- [Cells: Backups](cells-feature-backups.md) -- [Cells: CI Runners](cells-feature-ci-runners.md) -- [Cells: Container Registry](cells-feature-container-registry.md) -- [Cells: Contributions: Forks](cells-feature-contributions-forks.md) -- [Cells: Database Sequences](cells-feature-database-sequences.md) -- [Cells: Data Migration](cells-feature-data-migration.md) -- [Cells: Explore](cells-feature-explore.md) -- [Cells: Git Access](cells-feature-git-access.md) -- [Cells: Global Search](cells-feature-global-search.md) -- [Cells: GraphQL](cells-feature-graphql.md) -- [Cells: Organizations](cells-feature-organizations.md) -- [Cells: Secrets](cells-feature-secrets.md) -- [Cells: Snippets](cells-feature-snippets.md) -- [Cells: User Profile](cells-feature-user-profile.md) -- [Cells: Your Work](cells-feature-your-work.md) +- [Cells: Admin Area](impacted_features/admin-area.md) +- [Cells: Backups](impacted_features/backups.md) +- [Cells: CI Runners](impacted_features/ci-runners.md) +- [Cells: Container Registry](impacted_features/container-registry.md) +- [Cells: Contributions: Forks](impacted_features/contributions-forks.md) +- [Cells: Database Sequences](impacted_features/database-sequences.md) +- [Cells: Data Migration](impacted_features/data-migration.md) +- [Cells: Explore](impacted_features/explore.md) +- [Cells: Git Access](impacted_features/git-access.md) +- [Cells: Global Search](impacted_features/global-search.md) +- [Cells: GraphQL](impacted_features/graphql.md) +- [Cells: Organizations](impacted_features/organizations.md) +- [Cells: Personal Namespaces](impacted_features/personal-namespaces.md) +- [Cells: Secrets](impacted_features/secrets.md) +- [Cells: Snippets](impacted_features/snippets.md) +- [Cells: User Profile](impacted_features/user-profile.md) +- [Cells: Your Work](impacted_features/your-work.md) ### Impacted features: Placeholders The following list of impacted features only represents placeholders that still require work to estimate the impact of Cells and develop solution proposals. -- [Cells: Agent for Kubernetes](cells-feature-agent-for-kubernetes.md) -- [Cells: GitLab Pages](cells-feature-gitlab-pages.md) -- [Cells: Personal Access Tokens](cells-feature-personal-access-tokens.md) -- [Cells: Personal Namespaces](cells-feature-personal-namespaces.md) -- [Cells: Router Endpoints Classification](cells-feature-router-endpoints-classification.md) -- [Cells: Schema changes (Postgres and Elasticsearch migrations)](cells-feature-schema-changes.md) -- [Cells: Uploads](cells-feature-uploads.md) +- [Cells: Agent for Kubernetes](impacted_features/agent-for-kubernetes.md) +- [Cells: CI/CD Catalog](impacted_features/ci-cd-catalog.md) +- [Cells: Data pipeline ingestion](impacted_features/data-pipeline-ingestion.md) +- [Cells: GitLab Pages](impacted_features/gitlab-pages.md) +- [Cells: Personal Access Tokens](impacted_features/personal-access-tokens.md) +- [Cells: Router Endpoints Classification](impacted_features/router-endpoints-classification.md) +- [Cells: Schema changes (Postgres and Elasticsearch migrations)](impacted_features/schema-changes.md) +- [Cells: Uploads](impacted_features/uploads.md) - ... ## Frequently Asked Questions @@ -367,6 +369,59 @@ For example, users on GitLab Dedicated don't have to have a different and unique Up until iteration 3, Cells communicate with each other only via a shared database that contains common data. In iteration 4 we are going to evaluate the option of Cells calling each other via API to provide more isolation and reliability. +### How are Cells provisioned? + +The GitLab.com cluster of Cells will use GitLab Dedicated instances. +Once a GitLab Dedicated instance gets provisioned it could join the GitLab.com cluster and become a Cell. +One requirement will be that the GitLab Dedicated instance does not contain any prior data. + +To reach shared resources, Cells will use [Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect). + +See also the [design discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/396641). + +### What is a Cells topology? + +See the [design discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/396641). + +### How are users of an Organization routed to the correct Cell? + +TBD + +### How do users authenticate with Cells and Organizations? + +See the [design discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/395736). + +### How are Cells rebalanced? + +TBD + +### How can Cells implement disaster recovery capabilities? + +TBD + +### How do I decide whether to move my feature to the cluster, Cell or Organization level? + +By default, features are required to be scoped to the Organization level. Any deviation from that rule should be validated and approved by Tenant Scale. + +The design goals of the Cells architecture describe that [all Cells are under a single domain](goals.md#all-cells-are-under-a-single-gitlabcom-domain) and as such, Cells are invisible to the user: + +- Cell-local features should be limited to those related to managing the Cell, but never be a feature where the Cell semantic is exposed to the customer. +- The Cells architecture wants to freely control the distribution of Organization and customer data across Cells without impacting users when data is migrated. + +Cluster-wide features are strongly discouraged because: + +- They might require storing a substantial amount of data cluster-wide which decreases [scalability headroom](goals.md#provides-100x-headroom). +- They might require implementation of non-trivial [data aggregation](goals.md#aggregation-of-cluster-wide-data) that reduces resilience to [single node failure](goals.md#high-resilience-to-a-single-cell-failure). +- They are harder to build due to the need of being able to run [mixed deployments](goals.md#cells-running-in-mixed-deployments). Cluster-wide features need to take this into account. +- They might affect our ability to provide an [on-premise like experience on GitLab.com](goals.md#on-premise-like-experience). +- Some features that are expected to be cluster-wide might in fact be better implemented using federation techniques that use trusted intra-cluster communication using the same user identity. User Profile is shared across the cluster. +- The Cells architecture limits what services can be considered cluster-wide. Services that might initially be cluster-wide are still expected to be split in the future to achieve full service isolation. No feature should be built to depend on such a service (like Elasticsearch). + +### Will Cells use the [reference architecture for 50,000 users](../../../administration/reference_architectures/50k_users.md)? + +The infrastructure team will properly size Cells depending on the load. +The Tenant Scale team sees an opportunity to use GitLab Dedicated as a base for Cells deployment. + ## Decision log - 2022-03-15: Google Cloud as the cloud service. For details, see [issue 396641](https://gitlab.com/gitlab-org/gitlab/-/issues/396641#note_1314932272). @@ -378,3 +433,4 @@ In iteration 4 we are going to evaluate the option of Cells calling each other v - [Database group investigation](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/database/doc/root-namespace-sharding.html) - [Shopify Pods architecture](https://shopify.engineering/a-pods-architecture-to-allow-shopify-to-scale) - [Opstrace architecture](https://gitlab.com/gitlab-org/opstrace/opstrace/-/blob/main/docs/architecture/overview.md) +- [Adding Diagrams to this blueprint](diagrams/index.md) diff --git a/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md b/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md index c1ca0c60dcd..847532a36dc 100644 --- a/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md +++ b/doc/architecture/blueprints/cells/proposal-stateless-router-with-buffering-requests.md @@ -27,10 +27,10 @@ monolith. This architecture also supports regions by allowing for low traffic databases to be replicated across regions. Users are not directly exposed to the concept of Cells but instead they see -different data dependent on their chosen "organization". -[Organizations](glossary.md#organizations) will be a new model introduced to enforce isolation in the -application and allow us to decide which request route to which cell, since an -organization can only be on a single cell. +different data dependent on their chosen Organization. +[Organizations](goals.md#organizations) will be a new entity introduced to enforce isolation in the +application and allow us to decide which request routes to which Cell, since an +Organization can only be on a single Cell. ## Differences diff --git a/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md b/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md index 3b3d481914f..962f71673df 100644 --- a/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md +++ b/doc/architecture/blueprints/cells/proposal-stateless-router-with-routes-learning.md @@ -27,10 +27,10 @@ monolith. This architecture also supports regions by allowing for low traffic databases to be replicated across regions. Users are not directly exposed to the concept of Cells but instead they see -different data dependent on their chosen "organization". -[Organizations](glossary.md#organizations) will be a new model introduced to enforce isolation in the -application and allow us to decide which request route to which cell, since an -organization can only be on a single cell. +different data dependent on their chosen Organization. +[Organizations](goals.md#organizations) will be a new entity introduced to enforce isolation in the +application and allow us to decide which request routes to which Cell, since an +Organization can only be on a single Cell. ## Differences |
