diff options
author | Ciro Santilli <ciro.santilli@gmail.com> | 2014-04-25 02:48:22 +0400 |
---|---|---|
committer | Ciro Santilli <ciro.santilli@gmail.com> | 2014-06-04 01:16:31 +0400 |
commit | fd348de76d651d49acc8eb742cc647dc777ef5fc (patch) | |
tree | 26f42bef57c9a636eff0a548a29cb1e2e6d12c8c /doc/development | |
parent | de1a7aa7eb523cf2fdad12f8eeda2ba4c5b51820 (diff) |
Update docs to markdown style guide.
Diffstat (limited to 'doc/development')
-rw-r--r-- | doc/development/README.md | 8 | ||||
-rw-r--r-- | doc/development/architecture.md | 119 | ||||
-rw-r--r-- | doc/development/shell_commands.md | 22 |
3 files changed, 55 insertions, 94 deletions
diff --git a/doc/development/README.md b/doc/development/README.md index eb88b6c860f..67ee828db6b 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -1,5 +1,5 @@ -## Development +# Development -+ [Architecture](architecture.md) of GitLab -+ [Shell commands](shell_commands.md) in the GitLab codebase -+ [Rake tasks](rake_tasks.md) for development +- [Architecture](architecture.md) of GitLab +- [Shell commands](shell_commands.md) in the GitLab codebase +- [Rake tasks](rake_tasks.md) for development diff --git a/doc/development/architecture.md b/doc/development/architecture.md index f2f7a3d5f0f..9d0d58b3db9 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -1,78 +1,54 @@ # GitLab Architecture Overview ---- -# Software delivery +## Software delivery -There are two editions of GitLab: [Enterprise Edition](https://www.gitlab.com/gitlab-ee/) (EE) and [Community Edition](https://www.gitlab.com/gitlab-ce/) (CE). -GitLab CE is delivered via git from the [gitlabhq repository](https://gitlab.com/gitlab-org/gitlab-ce/tree/master). -New versions of GitLab are released in stable branches and the master branch is for bleeding edge development. +There are two editions of GitLab: [Enterprise Edition](https://www.gitlab.com/gitlab-ee/) (EE) and [Community Edition](https://www.gitlab.com/gitlab-ce/) (CE). GitLab CE is delivered via git from the [gitlabhq repository](https://gitlab.com/gitlab-org/gitlab-ce/tree/master). New versions of GitLab are released in stable branches and the master branch is for bleeding edge development. -EE releases are available not long after CE releases. -To obtain the GitLab EE there is a [repository at gitlab.com](https://gitlab.com/subscribers/gitlab-ee). -For more information about the release process see the section 'New versions and upgrading' in the readme. +EE releases are available not long after CE releases. To obtain the GitLab EE there is a [repository at gitlab.com](https://gitlab.com/subscribers/gitlab-ee). For more information about the release process see the section 'New versions and upgrading' in the readme. -Both EE and CE require an add-on component called gitlab-shell. -It is obtained from the [gitlab-shell repository](https://gitlab.com/gitlab-org/gitlab-shell/tree/master). -New versions are usually tags but staying on the master branch will give you the latest stable version. -New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical. +Both EE and CE require an add-on component called gitlab-shell. It is obtained from the [gitlab-shell repository](https://gitlab.com/gitlab-org/gitlab-shell/tree/master). New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical. -# System Layout +## System Layout When referring to ~git in the pictures it means the home directory of the git user which is typically /home/git. -GitLab is primarily installed within the `/home/git` user home directory as `git` user. -Within the home directory is where the gitlabhq server software resides as well as the repositories (though the repository location is configurable). -The bare repositories are located in `/home/git/repositories`. -GitLab is a ruby on rails application so the particulars of the inner workings can be learned by studying how a ruby on rails application works. +GitLab is primarily installed within the `/home/git` user home directory as `git` user. Within the home directory is where the gitlabhq server software resides as well as the repositories (though the repository location is configurable). + +The bare repositories are located in `/home/git/repositories`. GitLab is a ruby on rails application so the particulars of the inner workings can be learned by studying how a ruby on rails application works. + To serve repositories over SSH there's an add-on application called gitlab-shell which is installed in `/home/git/gitlab-shell`. -## Components +### Components ![GitLab Diagram Overview](gitlab_diagram_overview.png) -A typical install of GitLab will be on Ubuntu Linux or RHEL/CentOS. -It uses Nginx or Apache as a web front end to proxypass the Unicorn web server. -By default, communication between Unicorn and the front end is via a Unix domain socket but forwarding requests via TCP is also supported. -The web front end accesses `/home/git/gitlab/public` bypassing the Unicorn server to serve static pages, uploads (e.g. avatar images or attachments), and precompiled assets. -GitLab serves web pages and a [GitLab API](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/api) using the Unicorn web server. -It uses Sidekiq as a job queue which, in turn, uses redis as a non-persistent database backend for job information, meta data, and incomming jobs. -The GitLab web app uses MySQL or PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). -GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. -It also keeps default branch and hook information with the bare repository. -`/home/git/gitlab-satellites` keeps checked out repositories when performing actions such as a merge request, editing files in the web interface, etc. -The satellite repository is used by the web interface for editing repositories and the wiki which is also a git repository. -When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving git objects. - -The add-on component gitlab-shell serves repositories over SSH. -It manages the SSH keys within `/home/git/.ssh/authorized_keys` which should not be manually edited. -gitlab-shell accesses the bare repositories directly to serve git objects and communicates with redis to submit jobs to Sidekiq for GitLab to process. - gitlab-shell queries the GitLab API to determine authorization and access. - -## Installation Folder Summary +A typical install of GitLab will be on Ubuntu Linux or RHEL/CentOS. It uses Nginx or Apache as a web front end to proxypass the Unicorn web server. By default, communication between Unicorn and the front end is via a Unix domain socket but forwarding requests via TCP is also supported. The web front end accesses `/home/git/gitlab/public` bypassing the Unicorn server to serve static pages, uploads (e.g. avatar images or attachments), and precompiled assets. GitLab serves web pages and a [GitLab API](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/api) using the Unicorn web server. It uses Sidekiq as a job queue which, in turn, uses redis as a non-persistent database backend for job information, meta data, and incomming jobs. -To summarize here's the [directory structure of the `git` user home directory](../install/structure.md). +The GitLab web app uses MySQL or PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository. `/home/git/gitlab-satellites` keeps checked out repositories when performing actions such as a merge request, editing files in the web interface, etc. + +The satellite repository is used by the web interface for editing repositories and the wiki which is also a git repository. When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving git objects. +The add-on component gitlab-shell serves repositories over SSH. It manages the SSH keys within `/home/git/.ssh/authorized_keys` which should not be manually edited. gitlab-shell accesses the bare repositories directly to serve git objects and communicates with redis to submit jobs to Sidekiq for GitLab to process. gitlab-shell queries the GitLab API to determine authorization and access. + +### Installation Folder Summary + +To summarize here's the [directory structure of the `git` user home directory](../install/structure.md). -## Processes +### Processes ps aux | grep '^git' -GitLab has several components to operate. -As a system user (i.e. any user that is not the `git` user) it requires a persistent database (MySQL/PostreSQL) and redis database. -It also uses Apache httpd or nginx to proxypass Unicorn. -As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server running on port `8080` by default). -Under the gitlab user there are normally 4 processes: `unicorn_rails master` (1 process), `unicorn_rails worker` (2 processes), `sidekiq` (1 process). +GitLab has several components to operate. As a system user (i.e. any user that is not the `git` user) it requires a persistent database (MySQL/PostreSQL) and redis database. It also uses Apache httpd or nginx to proxypass Unicorn. As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server running on port `8080` by default). Under the gitlab user there are normally 4 processes: `unicorn_rails master` (1 process), `unicorn_rails worker` (2 processes), `sidekiq` (1 process). -## Repository access +### Repository access -Repositories get accessed via HTTP or SSH. -HTTP cloning/push/pull utilizes the GitLab API and SSH cloning is handled by gitlab-shell (previously explained). +Repositories get accessed via HTTP or SSH. HTTP cloning/push/pull utilizes the GitLab API and SSH cloning is handled by gitlab-shell (previously explained). -# Troubleshooting +## Troubleshooting See the README for more information. -## Init scripts of the services +### Init scripts of the services The GitLab init script starts and stops Unicorn and Sidekiq. @@ -115,61 +91,59 @@ $ /etc/init.d/postgresql Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [version ..] ``` -## Log locations of the services +### Log locations of the services Note: `/home/git/` is shorthand for `/home/git`. gitlabhq (includes Unicorn and Sidekiq logs) -* `/home/git/gitlab/log/` contains `application.log`, `production.log`, `sidekiq.log`, `unicorn.stdout.log`, `githost.log`, `satellites.log`, and `unicorn.stderr.log` normally. +- `/home/git/gitlab/log/` contains `application.log`, `production.log`, `sidekiq.log`, `unicorn.stdout.log`, `githost.log`, `satellites.log`, and `unicorn.stderr.log` normally. gitlab-shell -* `/home/git/gitlab-shell/gitlab-shell.log` +- `/home/git/gitlab-shell/gitlab-shell.log` ssh -* `/var/log/auth.log` auth log (on Ubuntu). -* `/var/log/secure` auth log (on RHEL). +- `/var/log/auth.log` auth log (on Ubuntu). +- `/var/log/secure` auth log (on RHEL). nginx -* `/var/log/nginx/` contains error and access logs. +- `/var/log/nginx/` contains error and access logs. Apache httpd -* [Explanation of apache logs](http://httpd.apache.org/docs/2.2/logs.html). -* `/var/log/apache2/` contains error and output logs (on Ubuntu). -* `/var/log/httpd/` contains error and output logs (on RHEL). +- [Explanation of apache logs](http://httpd.apache.org/docs/2.2/logs.html). +- `/var/log/apache2/` contains error and output logs (on Ubuntu). +- `/var/log/httpd/` contains error and output logs (on RHEL). redis -* `/var/log/redis/redis.log` there are also logrotated logs there. +- `/var/log/redis/redis.log` there are also logrotated logs there. PostgreSQL -* `/var/log/postgresql/*` +- `/var/log/postgresql/*` MySQL -* `/var/log/mysql/*` -* `/var/log/mysql.*` +- `/var/log/mysql/*` +- `/var/log/mysql.*` -## GitLab specific config files +### GitLab specific config files -GitLab has configuration files located in `/home/git/gitlab/config/*`. -Commonly referenced config files include: +GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced config files include: -* `gitlab.yml` - GitLab configuration. -* `unicorn.rb` - Unicorn web server settings. -* `database.yml` - Database connection settings. +- `gitlab.yml` - GitLab configuration. +- `unicorn.rb` - Unicorn web server settings. +- `database.yml` - Database connection settings. gitlab-shell has a configuration file at `/home/git/gitlab-shell/config.yml`. -## Maintenance Tasks +### Maintenance Tasks -[GitLab](https://gitlab.com/gitlab-org/gitlab-ce/tree/master) provides rake tasks with which you see version information and run a quick check on your configuration to ensure it is configured properly within the application. -See [maintenance rake tasks](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/raketasks/maintenance.md). +[GitLab](https://gitlab.com/gitlab-org/gitlab-ce/tree/master) provides rake tasks with which you see version information and run a quick check on your configuration to ensure it is configured properly within the application. See [maintenance rake tasks](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/raketasks/maintenance.md). In a nutshell, do the following: ``` @@ -179,5 +153,4 @@ bundle exec rake gitlab:env:info RAILS_ENV=production bundle exec rake gitlab:check RAILS_ENV=production ``` -Note: It is recommended to log into the `git` user using `sudo -i -u git` or `sudo su - git`. -While the sudo commands provided by gitlabhq work in Ubuntu they do not always work in RHEL.
\ No newline at end of file +Note: It is recommended to log into the `git` user using `sudo -i -u git` or `sudo su - git`. While the sudo commands provided by gitlabhq work in Ubuntu they do not always work in RHEL. diff --git a/doc/development/shell_commands.md b/doc/development/shell_commands.md index af0d5ca4426..1f3908f4e27 100644 --- a/doc/development/shell_commands.md +++ b/doc/development/shell_commands.md @@ -8,9 +8,7 @@ ## Use File and FileUtils instead of shell commands -Sometimes we invoke basic Unix commands via the shell when there is also a Ruby API for doing it. -Use the Ruby API if it exists. -http://www.ruby-doc.org/stdlib-2.0.0/libdoc/fileutils/rdoc/FileUtils.html#module-FileUtils-label-Module+Functions +Sometimes we invoke basic Unix commands via the shell when there is also a Ruby API for doing it. Use the Ruby API if it exists. <http://www.ruby-doc.org/stdlib-2.0.0/libdoc/fileutils/rdoc/FileUtils.html#module-FileUtils-label-Module+Functions> ```ruby # Wrong @@ -30,12 +28,7 @@ This coding style could have prevented CVE-2013-4490. ## Bypass the shell by splitting commands into separate tokens -When we pass shell commands as a single string to Ruby, Ruby will let `/bin/sh` evaluate the entire string. -Essentially, we are asking the shell to evaluate a one-line script. -This creates a risk for shell injection attacks. -It is better to split the shell command into tokens ourselves. -Sometimes we use the scripting capabilities of the shell to change the working directory or set environment variables. -All of this can also be achieved securely straight from Ruby +When we pass shell commands as a single string to Ruby, Ruby will let `/bin/sh` evaluate the entire string. Essentially, we are asking the shell to evaluate a one-line script. This creates a risk for shell injection attacks. It is better to split the shell command into tokens ourselves. Sometimes we use the scripting capabilities of the shell to change the working directory or set environment variables. All of this can also be achieved securely straight from Ruby ```ruby # Wrong @@ -55,8 +48,7 @@ This coding style could have prevented CVE-2013-4546. ## Separate options from arguments with -- -Make the difference between options and arguments clear to the argument parsers of system commands with `--`. -This is supported by many but not all Unix commands. +Make the difference between options and arguments clear to the argument parsers of system commands with `--`. This is supported by many but not all Unix commands. To understand what `--` does, consider the problem below. @@ -68,9 +60,7 @@ cat: illegal option -- l usage: cat [-benstuv] [file ...] ``` -In the example above, the argument parser of `cat` assumes that `-l` is an option. -The solution in the example above is to make it clear to `cat` that `-l` is really an argument, not an option. -Many Unix command line tools follow the convention of separating options from arguments with `--`. +In the example above, the argument parser of `cat` assumes that `-l` is an option. The solution in the example above is to make it clear to `cat` that `-l` is really an argument, not an option. Many Unix command line tools follow the convention of separating options from arguments with `--`. ``` # Example (continued) @@ -91,9 +81,7 @@ This coding style could have prevented CVE-2013-4582. ## Do not use the backticks -Capturing the output of shell commands with backticks reads nicely, but you are forced to pass the command as one string to the shell. -We explained above that this is unsafe. -In the main GitLab codebase, the solution is to use `Gitlab::Popen.popen` instead. +Capturing the output of shell commands with backticks reads nicely, but you are forced to pass the command as one string to the shell. We explained above that this is unsafe. In the main GitLab codebase, the solution is to use `Gitlab::Popen.popen` instead. ```ruby # Wrong |