diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-10-05 16:54:15 +0300 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-10-05 16:54:15 +0300 |
commit | be834a25982746ffd85252ff502df42bb88cb9d5 (patch) | |
tree | b4d6a8ba0931e12fac08f05abea33a3b8ec2c8a2 /doc/development/architecture.md | |
parent | ee925a3597f27e92f83a50937a64068109675b3d (diff) |
Add latest changes from gitlab-org/gitlab@13-5-stable-eev13.5.0-rc32
Diffstat (limited to 'doc/development/architecture.md')
-rw-r--r-- | doc/development/architecture.md | 196 |
1 files changed, 140 insertions, 56 deletions
diff --git a/doc/development/architecture.md b/doc/development/architecture.md index f6b1c8cd914..2646ad7052b 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -1,32 +1,91 @@ -# GitLab Architecture Overview +# GitLab architecture overview ## Software delivery -There are two software distributions of GitLab: the open source [Community Edition](https://gitlab.com/gitlab-org/gitlab-foss/) (CE), and the open core [Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/) (EE). GitLab is available under [different subscriptions](https://about.gitlab.com/pricing/). +There are two software distributions of GitLab: -New versions of GitLab are released in stable branches and the master branch is for bleeding edge development. +- The open source [Community Edition](https://gitlab.com/gitlab-org/gitlab-foss/) (CE). +- The open core [Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/) (EE). -For information, see the [GitLab Release Process](https://gitlab.com/gitlab-org/release/docs/-/tree/master#gitlab-release-process). +GitLab is available under [different subscriptions](https://about.gitlab.com/pricing/). -Both EE and CE require some add-on components called GitLab Shell and Gitaly. These components are available from the [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/master) and [Gitaly](https://gitlab.com/gitlab-org/gitaly/-/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with the exception of informal security updates deemed critical. +New versions of GitLab are released from stable branches, and the `master` branch is used for +bleeding-edge development. + +For more information, visit the [GitLab Release Process](https://about.gitlab.com/handbook/engineering/releases/). + +Both distributions require additional components. These components are described in the +[Component details](#components) section, and all have their own repositories. +New versions of each dependent component are usually tags, but staying on the `master` branch of the +GitLab codebase gives you the latest stable version of those components. New versions are +generally released around the same time as GitLab releases, with the exception of informal security +updates deemed critical. ## Components -A typical install of GitLab will be on GNU/Linux. It uses NGINX or Apache as a web front end to proxypass the Unicorn web server. By default, communication between Unicorn and the front end is via a Unix domain socket but forwarding requests via TCP is also supported. The web front end accesses `/home/git/gitlab/public` bypassing the Unicorn server to serve static pages, uploads (e.g. avatar images or attachments), and pre-compiled assets. GitLab serves web pages and the [GitLab API](../api/README.md) using the Unicorn web server. It uses Sidekiq as a job queue which, in turn, uses Redis as a non-persistent database backend for job information, meta data, and incoming jobs. +A typical install of GitLab is on GNU/Linux, but growing number of deployments also use the +Kubernetes platform. The largest known GitLab instance is on GitLab.com, which is deployed using our +[official GitLab Helm chart](https://docs.gitlab.com/charts/) and the [official Linux package](https://about.gitlab.com/install/). + +A typical installation uses NGINX or Apache as a web server to proxy through +[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) and into the [Puma](https://puma.io) +application server. GitLab serves web pages and the [GitLab API](../api/README.md) using the Puma +application server. It uses Sidekiq as a job queue which, in turn, uses Redis as a non-persistent +database backend for job information, metadata, and incoming jobs. -We also support deploying GitLab on Kubernetes using our [GitLab Helm chart](https://docs.gitlab.com/charts/). +By default, communication between Puma and Workhorse is via a Unix domain socket, but forwarding +requests via TCP is also supported. Workhorse accesses the `gitlab/public` directory, bypassing the +Puma application server to serve static pages, uploads (for example, avatar images or attachments), +and pre-compiled assets. -The GitLab web app uses PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare Git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository. +The GitLab application uses PostgreSQL for persistent database information (for example, users, +permissions, issues, or other metadata). GitLab stores the bare Git repositories in the location +defined in [the configuration file, `repositories:` section](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example). +It also keeps default branch and hook information with the bare repository. -When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving Git objects. +When serving repositories over HTTP/HTTPS GitLab uses the GitLab API to resolve authorization and +access and to serve Git objects. -The add-on component GitLab Shell serves repositories over SSH. It manages the SSH keys within `/home/git/.ssh/authorized_keys` which should not be manually edited. GitLab Shell accesses the bare repositories through Gitaly to serve Git objects and communicates with Redis to submit jobs to Sidekiq for GitLab to process. GitLab Shell queries the GitLab API to determine authorization and access. +The add-on component GitLab Shell serves repositories over SSH. It manages the SSH keys within the +location defined in [the configuration file, `GitLab Shell` section](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example). +The file in that location should never be manually edited. GitLab Shell accesses the bare +repositories through Gitaly to serve Git objects, and communicates with Redis to submit jobs to +Sidekiq for GitLab to process. GitLab Shell queries the GitLab API to determine authorization and access. -Gitaly executes Git operations from GitLab Shell and the GitLab web app, and provides an API to the GitLab web app to get attributes from Git (e.g. title, branches, tags, other meta data), and to get blobs (e.g. diffs, commits, files). +Gitaly executes Git operations from GitLab Shell and the GitLab web app, and provides an API to the +GitLab web app to get attributes from Git (for example, title, branches, tags, or other metadata), +and to get blobs (for example, diffs, commits, or files). You may also be interested in the [production architecture of GitLab.com](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/). -### Simplified Component Overview +## Adapting existing and introducing new components + +There are fundamental differences in how the application behaves when it is installed on a +traditional Linux machine compared to a containerized platform, such as Kubernetes. + +Compared to [our official installation methods](https://about.gitlab.com/install/), some of the +notable differences are: + +- Official Linux packages can access files on the same file system with different services. + [Shared files](shared_files.md) are not an option for the application running on the Kubernetes + platform. +- Official Linux packages by default have services that have access to the shared configuration and + network. This is not the case for services running in Kubernetes, where services might be running + in complete isolation, or only accessible through specific ports. + +In other words, the shared state between services needs to be carefully considered when +architecting new features and adding new components. Services that need to have access to the same +files, need to be able to exchange information through the appropriate APIs. Whenever possible, +this should not be done with files. + +Since components written with the API-first philosophy in mind are compatible with both methods, all +new features and services must be written to consider Kubernetes compatibility **first**. + +The simplest way to ensure this, is to add support for your feature or service to +[the official GitLab Helm chart](https://docs.gitlab.com/charts/) or reach out to +[the Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/distribution/#how-to-work-with-distribution). + +### Simplified component overview This is a simplified architecture diagram that can be used to understand GitLab's architecture. @@ -51,15 +110,15 @@ SSH -- TCP 22 --> GitLabShell[GitLab Shell] SMTP[SMTP Gateway] Geo[GitLab Geo Node] -- TCP 22, 80, 443 --> NGINX -GitLabShell --TCP 8080 -->Unicorn["Unicorn (GitLab Rails)"] +GitLabShell --TCP 8080 -->Puma["Puma (GitLab Rails)"] GitLabShell --> Praefect -Unicorn --> PgBouncer[PgBouncer] -Unicorn --> Redis -Unicorn --> Praefect +Puma --> PgBouncer[PgBouncer] +Puma --> Redis +Puma --> Praefect Sidekiq --> Redis Sidekiq --> PgBouncer Sidekiq --> Praefect -GitLabWorkhorse[GitLab Workhorse] --> Unicorn +GitLabWorkhorse[GitLab Workhorse] --> Puma GitLabWorkhorse --> Redis GitLabWorkhorse --> Praefect Praefect --> Gitaly @@ -67,7 +126,7 @@ NGINX --> GitLabWorkhorse NGINX -- TCP 8090 --> GitLabPages[GitLab Pages] NGINX --> Grafana[Grafana] Grafana -- TCP 9090 --> Prometheus[Prometheus] -Prometheus -- TCP 80, 443 --> Unicorn +Prometheus -- TCP 80, 443 --> Puma RedisExporter[Redis Exporter] --> Redis Prometheus -- TCP 9121 --> RedisExporter PostgreSQLExporter[PostgreSQL Exporter] --> PostgreSQL @@ -83,27 +142,27 @@ PgBouncer --> Consul PostgreSQL --> Consul PgBouncer --> PostgreSQL NGINX --> Registry -Unicorn --> Registry +Puma --> Registry NGINX --> Mattermost -Mattermost --- Unicorn +Mattermost --- Puma Prometheus --> Alertmanager Migrations --> PostgreSQL Runner -- TCP 443 --> NGINX -Unicorn -- TCP 9200 --> Elasticsearch +Puma -- TCP 9200 --> Elasticsearch Sidekiq -- TCP 9200 --> Elasticsearch Sidekiq -- TCP 80, 443 --> Sentry -Unicorn -- TCP 80, 443 --> Sentry +Puma -- TCP 80, 443 --> Sentry Sidekiq -- UDP 6831 --> Jaeger -Unicorn -- UDP 6831 --> Jaeger +Puma -- UDP 6831 --> Jaeger Gitaly -- UDP 6831 --> Jaeger GitLabShell -- UDP 6831 --> Jaeger GitLabWorkhorse -- UDP 6831 --> Jaeger Alertmanager -- TCP 25 --> SMTP Sidekiq -- TCP 25 --> SMTP -Unicorn -- TCP 25 --> SMTP -Unicorn -- TCP 369 --> LDAP +Puma -- TCP 25 --> SMTP +Puma -- TCP 369 --> LDAP Sidekiq -- TCP 369 --> LDAP -Unicorn -- TCP 443 --> ObjectStorage["Object Storage"] +Puma -- TCP 443 --> ObjectStorage["Object Storage"] Sidekiq -- TCP 443 --> ObjectStorage GitLabWorkhorse -- TCP 443 --> ObjectStorage Registry -- TCP 443 --> ObjectStorage @@ -121,7 +180,7 @@ click Gitaly "./architecture.html#gitaly" click Jaeger "./architecture.html#jaeger" click GitLabWorkhorse "./architecture.html#gitlab-workhorse" click LDAP "./architecture.html#ldap-authentication" -click Unicorn "./architecture.html#unicorn" +click Puma "./architecture.html#puma" click GitLabShell "./architecture.html#gitlab-shell" click SSH "./architecture.html#ssh-request-22" click Sidekiq "./architecture.html#sidekiq" @@ -201,7 +260,7 @@ Table description links: | [Runner](#gitlab-runner) | Executes GitLab CI/CD jobs | ⤓ | ✅ | ⚙ | ✅ | ⚙ | ⚙ | CE & EE | | [Sentry integration](#sentry) | Error tracking for deployed apps | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | CE & EE | | [Sidekiq](#sidekiq) | Background jobs processor | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | CE & EE | -| [Unicorn (GitLab Rails)](#unicorn) | Handles requests for the web interface and API | ✅ | ✅ | ✅ | ✅ | ⚙ | ✅ | CE & EE | +| [Puma (GitLab Rails)](#puma) | Handles requests for the web interface and API | ✅ | ✅ | ✅ | ✅ | ⚙ | ✅ | CE & EE | ### Component details @@ -271,7 +330,7 @@ Consul is a tool for service discovery and configuration. Consul is distributed, - [Source](../integration/elasticsearch.md) - [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/elasticsearch.md) - Layer: Core Service (Data) -- GitLab.com: [Get Advanced Global Search working on GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/153) epic. +- GitLab.com: [Get Advanced Search working on GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/153) epic. Elasticsearch is a distributed RESTful search engine built for the cloud. @@ -368,13 +427,13 @@ GitLab CI/CD is the open-source continuous integration service included with Git - [Project page](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/README.md) - Configuration: - [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template) - - [Charts](https://docs.gitlab.com/charts/charts/gitlab/unicorn/) + - [Charts](https://docs.gitlab.com/charts/charts/gitlab/webservice/) - [Source](../install/installation.md#install-gitlab-workhorse) - Layer: Core Service (Processor) - Process: `gitlab-workhorse` - GitLab.com: [Service Architecture](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#service-architecture) -[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alleviate pressure from Unicorn. You can read more about the [historical reasons for developing](https://about.gitlab.com/blog/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole. +[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alleviate pressure from Puma. You can read more about the [historical reasons for developing](https://about.gitlab.com/blog/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole. #### Grafana @@ -411,7 +470,8 @@ For monitoring deployed apps, see [Jaeger tracing documentation](../operations/t - Layer: Core Service - Process: `logrotate` -GitLab is comprised of a large number of services that all log. We started bundling our own logrotate as of 7.4 to make sure we were logging responsibly. This is just a packaged version of the common open source offering. +GitLab is comprised of a large number of services that all log. We started bundling our own Logrotate +as of GitLab 7.4 to make sure we were logging responsibly. This is just a packaged version of the common open source offering. #### Mattermost @@ -603,8 +663,30 @@ For monitoring deployed apps, see the [Sentry integration docs](../operations/er Sidekiq is a Ruby background job processor that pulls jobs from the Redis queue and processes them. Background jobs allow GitLab to provide a faster request/response cycle by moving work into the background. +#### Puma + +NOTE: **Note:** +Starting with GitLab 13.0, Puma is the default web server and Unicorn has been +disabled by default. + +- [Project page](https://gitlab.com/gitlab-org/gitlab/blob/master/README.md) +- Configuration: + - [Omnibus](https://docs.gitlab.com/omnibus/settings/puma.html) + - [Charts](https://docs.gitlab.com/charts/charts/gitlab/webservice/) + - [Source](../install/installation.md#configure-it) + - [GDK](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) +- Layer: Core Service (Processor) +- Process: `puma` +- GitLab.com: [Puma](../user/gitlab_com/index.md#puma) + +[Puma](https://puma.io/) is a Ruby application server that is used to run the core Rails Application that provides the user facing features in GitLab. Often process output you will see this as `bundle` or `config.ru` depending on the GitLab version. + #### Unicorn +NOTE: **Note:** +Starting with GitLab 13.0, Puma is the default web server and Unicorn has been +disabled by default. + - [Project page](https://gitlab.com/gitlab-org/gitlab/blob/master/README.md) - Configuration: - [Omnibus](https://docs.gitlab.com/omnibus/settings/unicorn.html) @@ -669,7 +751,7 @@ You can install them after you create a cluster. This includes: - [JupyterHub](https://jupyter.org) - [Knative](https://cloud.google.com/knative/) -## GitLab by Request Type +## GitLab by request type GitLab provides two "interfaces" for end users to access the service: @@ -678,20 +760,20 @@ GitLab provides two "interfaces" for end users to access the service: It's important to understand the distinction as some processes are used in both and others are exclusive to a specific request type. -### GitLab Web HTTP Request Cycle +### GitLab Web HTTP request cycle When making a request to an HTTP Endpoint (think `/users/sign_in`) the request will take the following path through the GitLab Service: - NGINX - Acts as our first line reverse proxy. -- GitLab Workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Unicorn. -- Unicorn - Since this is a web request, and it needs to access the application it will go to Unicorn. +- GitLab Workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Puma. +- Puma - Since this is a web request, and it needs to access the application it will go to Puma. - PostgreSQL/Gitaly/Redis - Depending on the type of request, it may hit these services to store or retrieve data. -### GitLab Git Request Cycle +### GitLab Git request cycle Below we describe the different paths that HTTP vs. SSH Git requests will take. There is some overlap with the Web Request Cycle but also some differences. -### Web Request (80/443) +### Web request (80/443) Git operations over HTTP use the stateless "smart" protocol described in the [Git documentation](https://git-scm.com/docs/http-protocol), but responsibility @@ -736,7 +818,7 @@ sequenceDiagram The sequence is similar for `git push`, except `git-receive-pack` is used instead of `git-upload-pack`. -### SSH Request (22) +### SSH request (22) Git operations over SSH can use the stateful protocol described in the [Git documentation](https://git-scm.com/docs/pack-protocol#_ssh_transport), but @@ -801,7 +883,7 @@ except there is no round-trip into Gitaly - Rails performs the action as part of the [internal API](internal_api.md) call, and GitLab Shell streams the response back to the user directly. -## System Layout +## System layout When referring to `~git` in the pictures it means the home directory of the Git user which is typically `/home/git`. @@ -811,9 +893,9 @@ The bare repositories are located in `/home/git/repositories`. GitLab is a Ruby To serve repositories over SSH there's an add-on application called GitLab Shell which is installed in `/home/git/gitlab-shell`. -### Installation Folder Summary +### Installation folder summary -To summarize here's the [directory structure of the `git` user home directory](../install/structure.md). +To summarize here's the [directory structure of the `git` user home directory](../install/installation.md#gitlab-directory-structure). ### Processes @@ -823,12 +905,12 @@ ps aux | grep '^git' GitLab has several components to operate. It requires a persistent database (PostgreSQL) and Redis database, and uses Apache `httpd` or NGINX to proxypass -Unicorn. All these components should run as different system users to GitLab -(e.g., `postgres`, `redis` and `www-data`, instead of `git`). +Puma. All these components should run as different system users to GitLab +(for example, `postgres`, `redis`, and `www-data`, instead of `git`). -As the `git` user it starts Sidekiq and Unicorn (a simple Ruby HTTP server +As the `git` user it starts Sidekiq and Puma (a simple Ruby HTTP server running on port `8080` by default). Under the GitLab user there are normally 4 -processes: `unicorn_rails master` (1 process), `unicorn_rails worker` +processes: `puma master` (1 process), `puma cluster worker` (2 processes), `sidekiq` (1 process). ### Repository access @@ -841,7 +923,7 @@ See the README for more information. ### Init scripts of the services -The GitLab init script starts and stops Unicorn and Sidekiq: +The GitLab init script starts and stops Puma and Sidekiq: ```plaintext /etc/init.d/gitlab @@ -881,9 +963,9 @@ Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [v ### Log locations of the services -GitLab (includes Unicorn and Sidekiq logs): +GitLab (includes Puma and Sidekiq logs): -- `/home/git/gitlab/log/` contains `application.log`, `production.log`, `sidekiq.log`, `unicorn.stdout.log`, `git_json.log` and `unicorn.stderr.log` normally. +- `/home/git/gitlab/log/` contains `application.log`, `production.log`, `sidekiq.log`, `puma.stdout.log`, `git_json.log` and `puma.stderr.log` normally. GitLab Shell: @@ -914,17 +996,18 @@ PostgreSQL: ### GitLab specific configuration files -GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced config files include: +GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced +configuration files include: -- `gitlab.yml` - GitLab configuration. -- `unicorn.rb` - Unicorn web server settings. -- `database.yml` - Database connection settings. +- `gitlab.yml` - GitLab configuration +- `puma.rb` - Puma web server settings +- `database.yml` - Database connection settings GitLab Shell has a configuration file at `/home/git/gitlab-shell/config.yml`. -### Maintenance Tasks +### Maintenance tasks -[GitLab](https://gitlab.com/gitlab-org/gitlab/tree/master) provides Rake tasks with which you see version information and run a quick check on your configuration to ensure it is configured properly within the application. See [maintenance Rake tasks](../raketasks/maintenance.md). +[GitLab](https://gitlab.com/gitlab-org/gitlab/tree/master) provides Rake tasks with which you see version information and run a quick check on your configuration to ensure it is configured properly within the application. See [maintenance Rake tasks](../administration/raketasks/maintenance.md). In a nutshell, do the following: ```shell @@ -934,7 +1017,8 @@ bundle exec rake gitlab:env:info RAILS_ENV=production bundle exec rake gitlab:check RAILS_ENV=production ``` -Note: It is recommended to log into the `git` user using `sudo -i -u git` or `sudo su - git`. While the sudo commands provided by GitLab work in Ubuntu they do not always work in RHEL. +Note: It is recommended to log into the `git` user using `sudo -i -u git` or `sudo su - git`. While +the `sudo` commands provided by GitLab work in Ubuntu they do not always work in RHEL. ## GitLab.com |