diff options
| author | Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com> | 2015-02-28 00:01:57 +0300 |
|---|---|---|
| committer | Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com> | 2015-02-28 00:01:57 +0300 |
| commit | 0d22b75b03496ced3d783f8fee9584098602ea1c (patch) | |
| tree | c7ddec6072c716fd63a8703f2dfeb0e4234a633f /doc | |
| parent | 5f682094d9b7c985ad62ebe29664bb6fe87b54be (diff) | |
| parent | d4aab6528cb80b0f41bdac2240dd9cc32543481d (diff) | |
Merge branch 'master' into mmonaco/gitlab-ce-api-user-noconfirm
Conflicts:
lib/api/users.rb
Diffstat (limited to 'doc')
130 files changed, 2790 insertions, 695 deletions
diff --git a/doc/README.md b/doc/README.md index 7343d5ae273..4e00dceac2b 100644 --- a/doc/README.md +++ b/doc/README.md @@ -2,26 +2,30 @@ ## User documentation -- [API](api/README.md) Explore how you can access GitLab via a simple and powerful API. -- [Markdown](markdown/markdown.md) Learn what you can do with GitLab's advanced formatting system. +- [API](api/README.md) Automate GitLab via a simple and powerful API. +- [Markdown](markdown/markdown.md) GitLab's advanced formatting system. - [Permissions](permissions/permissions.md) Learn what each role in a project (guest/reporter/developer/master/owner) can do. -- [Project Services](project_services/project_services.md) Explore how project services can integrate a project with external services, such as for CI. -- [Public access](public_access/public_access.md) Learn how you can allow public and internal access to a project. +- [Project Services](project_services/project_services.md) Integrate a project with external services, such as CI and chat. +- [Public access](public_access/public_access.md) Learn how you can allow public and internal access to projects. - [SSH](ssh/README.md) Setup your ssh keys and deploy keys for secure access to your projects. - [Web hooks](web_hooks/web_hooks.md) Let GitLab notify you when new code has been pushed to your project. -- [Workflow](workflow/README.md) Learn how to use Git and GitLab together. +- [Workflow](workflow/README.md) Using GitLab functionality and importing projects from GitHub and SVN. +- [GitLab as OAuth2 authentication service provider](integration/oauth_provider.md). It allows you to login to other applications from GitLab. ## Administrator documentation -- [Install](install/README.md) Requirements, directory structures and manual installation. +- [Install](install/README.md) Requirements, directory structures and installation from source. - [Integration](integration/README.md) How to integrate with systems such as JIRA, Redmine, LDAP and Twitter. -- [Raketasks](raketasks/README.md) Explore what GitLab has in store for you to make administration easier. -- [System hooks](system_hooks/system_hooks.md) Let GitLab notify you when certain management tasks need to be carried out. +- [Raketasks](raketasks/README.md) Backups, maintenance, automatic web hook setup and the importing of projects. +- [Custom git hooks](hooks/custom_hooks.md) Custom git hooks (on the filesystem) for when web hooks aren't enough. +- [System hooks](system_hooks/system_hooks.md) Notifications when users, projects and keys are changed. - [Security](security/README.md) Learn what you can do to further secure your GitLab instance. - [Update](update/README.md) Update guides to upgrade your installation. - [Welcome message](customization/welcome_message.md) Add a custom welcome message to the sign-in page. - [Issue closing](customization/issue_closing.md) Customize how to close an issue from commit messages. - [Libravatar](customization/libravatar.md) Use Libravatar for user avatars. +- [Operations](operations/README.md) Keeping GitLab up and running +- [Log system](logs/logs.md) Log system ## Contributor documentation diff --git a/doc/api/README.md b/doc/api/README.md index ffe250df3ff..dec530d0b81 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -22,6 +22,7 @@ ## Clients Find API Clients for GitLab [on our website](https://about.gitlab.com/applications/#api-clients). +You can use [GitLab as an OAuth2 client](oauth2.md) to make API calls. ## Introduction @@ -51,6 +52,24 @@ curl --header "PRIVATE-TOKEN: QVy1PB7sTxfy4pqfZM1U" "http://example.com/api/v3/p The API uses JSON to serialize data. You don't need to specify `.json` at the end of API URL. +## Authentication with OAuth2 token + +Instead of the private_token you can transmit the OAuth2 access token as a header or as a parameter. + +### OAuth2 token (as a parameter) + +``` +curl https://localhost:3000/api/v3/user?access_token=OAUTH-TOKEN +``` + +### OAuth2 token (as a header) + +``` +curl -H "Authorization: Bearer OAUTH-TOKEN" https://localhost:3000/api/v3/user +``` + +Read more about [GitLab as an OAuth2 client](oauth2.md). + ## Status codes The API is designed to return different status codes according to context and action. In this way if a request results in an error the caller is able to get insight into what went wrong, e.g. status code `400 Bad Request` is returned if a required attribute is missing from the request. The following list gives an overview of how the API functions generally behave. @@ -79,7 +98,7 @@ Return values: ## Sudo -All API requests support performing an api call as if you were another user, if your private token is for an administration account. You need to pass `sudo` parameter by url or header with an id or username of the user you want to perform the operation as. If passed as header, the header name must be "SUDO" (capitals). +All API requests support performing an api call as if you were another user, if your private token is for an administration account. You need to pass `sudo` parameter by URL or header with an id or username of the user you want to perform the operation as. If passed as header, the header name must be "SUDO" (capitals). If a non administrative `private_token` is provided then an error message will be returned with status code 403: @@ -124,7 +143,7 @@ When listing resources you can pass the following parameters: - `page` (default: `1`) - page number - `per_page` (default: `20`, max: `100`) - number of items to list per page -[Link headers](http://www.w3.org/wiki/LinkHeader) are send back with each response. These have `rel` prev/next/first/last and contain the relevant URL. Please use these instead of generating your own urls. +[Link headers](http://www.w3.org/wiki/LinkHeader) are send back with each response. These have `rel` prev/next/first/last and contain the relevant URL. Please use these instead of generating your own URLs. ## id vs iid diff --git a/doc/api/branches.md b/doc/api/branches.md index 319f0b47386..6a9c10c8520 100644 --- a/doc/api/branches.md +++ b/doc/api/branches.md @@ -15,27 +15,20 @@ Parameters: ```json [ { - "name": "master", "commit": { + "author_email": "john@example.com", + "author_name": "John Smith", + "authored_date": "2012-06-27T05:51:39-07:00", + "committed_date": "2012-06-28T03:44:20-07:00", + "committer_email": "john@example.com", + "committer_name": "John Smith", "id": "7b5c3cc8be40ee161ae89a06bba6229da1032a0c", - "parents": [ - { - "id": "4ad91d3c1144c406e50c7b33bae684bd6837faf8" - } - ], - "tree": "46e82de44b1061621357f24c05515327f2795a95", "message": "add projects API", - "author": { - "name": "John Smith", - "email": "john@example.com" - }, - "committer": { - "name": "John Smith", - "email": "john@example.com" - }, - "authored_date": "2012-06-27T05:51:39-07:00", - "committed_date": "2012-06-28T03:44:20-07:00" + "parent_ids": [ + "4ad91d3c1144c406e50c7b33bae684bd6837faf8" + ] }, + "name": "master", "protected": true } ] @@ -56,27 +49,20 @@ Parameters: ```json { - "name": "master", "commit": { + "author_email": "john@example.com", + "author_name": "John Smith", + "authored_date": "2012-06-27T05:51:39-07:00", + "committed_date": "2012-06-28T03:44:20-07:00", + "committer_email": "john@example.com", + "committer_name": "John Smith", "id": "7b5c3cc8be40ee161ae89a06bba6229da1032a0c", - "parents": [ - { - "id": "4ad91d3c1144c406e50c7b33bae684bd6837faf8" - } - ], - "tree": "46e82de44b1061621357f24c05515327f2795a95", "message": "add projects API", - "author": { - "name": "John Smith", - "email": "john@example.com" - }, - "committer": { - "name": "John Smith", - "email": "john@example.com" - }, - "authored_date": "2012-06-27T05:51:39-07:00", - "committed_date": "2012-06-28T03:44:20-07:00" + "parent_ids": [ + "4ad91d3c1144c406e50c7b33bae684bd6837faf8" + ] }, + "name": "master", "protected": true } ``` @@ -97,27 +83,20 @@ Parameters: ```json { - "name": "master", "commit": { + "author_email": "john@example.com", + "author_name": "John Smith", + "authored_date": "2012-06-27T05:51:39-07:00", + "committed_date": "2012-06-28T03:44:20-07:00", + "committer_email": "john@example.com", + "committer_name": "John Smith", "id": "7b5c3cc8be40ee161ae89a06bba6229da1032a0c", - "parents": [ - { - "id": "4ad91d3c1144c406e50c7b33bae684bd6837faf8" - } - ], - "tree": "46e82de44b1061621357f24c05515327f2795a95", "message": "add projects API", - "author": { - "name": "John Smith", - "email": "john@example.com" - }, - "committer": { - "name": "John Smith", - "email": "john@example.com" - }, - "authored_date": "2012-06-27T05:51:39-07:00", - "committed_date": "2012-06-28T03:44:20-07:00" + "parent_ids": [ + "4ad91d3c1144c406e50c7b33bae684bd6837faf8" + ] }, + "name": "master", "protected": true } ``` @@ -138,27 +117,20 @@ Parameters: ```json { - "name": "master", "commit": { + "author_email": "john@example.com", + "author_name": "John Smith", + "authored_date": "2012-06-27T05:51:39-07:00", + "committed_date": "2012-06-28T03:44:20-07:00", + "committer_email": "john@example.com", + "committer_name": "John Smith", "id": "7b5c3cc8be40ee161ae89a06bba6229da1032a0c", - "parents": [ - { - "id": "4ad91d3c1144c406e50c7b33bae684bd6837faf8" - } - ], - "tree": "46e82de44b1061621357f24c05515327f2795a95", "message": "add projects API", - "author": { - "name": "John Smith", - "email": "john@example.com" - }, - "committer": { - "name": "John Smith", - "email": "john@example.com" - }, - "authored_date": "2012-06-27T05:51:39-07:00", - "committed_date": "2012-06-28T03:44:20-07:00" + "parent_ids": [ + "4ad91d3c1144c406e50c7b33bae684bd6837faf8" + ] }, + "name": "master", "protected": false } ``` @@ -177,21 +149,20 @@ Parameters: ```json { - "name": "my-new-branch", "commit": { - "id": "8848c0e90327a0b70f1865b843fb2fbfb9345e57", - "message": "Merge pull request #54 from brightbox/use_fog_brightbox_module\n\nUpdate to use fog-brightbox module", - "parent_ids": [ - "fff449e0bf453576f16c91d6544f00a2664009d8", - "f93a93626fec20fd659f4ed3ab2e64019b6169ae" - ], - "authored_date": "2014-02-20T19:54:55+02:00", - "author_name": "john smith", "author_email": "john@example.com", - "committed_date": "2014-02-20T19:54:55+02:00", - "committer_name": "john smith", - "committer_email": "john@example.com" + "author_name": "John Smith", + "authored_date": "2012-06-27T05:51:39-07:00", + "committed_date": "2012-06-28T03:44:20-07:00", + "committer_email": "john@example.com", + "committer_name": "John Smith", + "id": "7b5c3cc8be40ee161ae89a06bba6229da1032a0c", + "message": "add projects API", + "parent_ids": [ + "4ad91d3c1144c406e50c7b33bae684bd6837faf8" + ] }, + "name": "master", "protected": false } ``` diff --git a/doc/api/commits.md b/doc/api/commits.md index 9475ecbaa67..eb8d6a43592 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -93,3 +93,66 @@ Parameters: } ] ``` + +## Get the comments of a commit + +Get the comments of a commit in a project. + +``` +GET /projects/:id/repository/commits/:sha/comments +``` + +Parameters: + +- `id` (required) - The ID of a project +- `sha` (required) - The name of a repository branch or tag or if not given the default branch + +```json +[ + { + "note": "this code is really nice", + "author": { + "id": 11, + "username": "admin", + "email": "admin@local.host", + "name": "Administrator", + "state": "active", + "created_at": "2014-03-06T08:17:35.000Z" + } + } +] +``` + +## Post comment to commit + +Adds a comment to a commit. Optionally you can post comments on a specific line of a commit. Therefor both `path`, `line_new` and `line_old` are required. + +``` +POST /projects/:id/repository/commits/:sha/comments +``` + +Parameters: + +- `id` (required) - The ID of a project +- `sha` (required) - The name of a repository branch or tag or if not given the default branch +- `note` (required) - Text of comment +- `path` (optional) - The file path +- `line` (optional) - The line number +- `line_type` (optional) - The line type (new or old) + +```json +{ + "author": { + "id": 1, + "username": "admin", + "email": "admin@local.host", + "name": "Administrator", + "blocked": false, + "created_at": "2012-04-29T08:46:00Z" + }, + "note": "text1", + "path": "example.rb", + "line": 5, + "line_type": "new" +} +``` diff --git a/doc/api/groups.md b/doc/api/groups.md index 6b379b02d28..b5a4b05ccaf 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -14,11 +14,13 @@ GET /groups "id": 1, "name": "Foobar Group", "path": "foo-bar", - "owner_id": 18 + "description": "An interesting group" } ] ``` +You can search for groups by name or path, see below. + ## Details of a group Get all details of a group. @@ -29,7 +31,7 @@ GET /groups/:id Parameters: -- `id` (required) - The ID of a group +- `id` (required) - The ID or path of a group ## New group @@ -43,6 +45,7 @@ Parameters: - `name` (required) - The name of the group - `path` (required) - The path of the group +- `description` (optional) - The group's description ## Transfer project to group @@ -54,7 +57,7 @@ POST /groups/:id/projects/:project_id Parameters: -- `id` (required) - The ID of a group +- `id` (required) - The ID or path of a group - `project_id` (required) - The ID of a project ## Remove group @@ -67,7 +70,26 @@ DELETE /groups/:id Parameters: -- `id` (required) - The ID of a user group +- `id` (required) - The ID or path of a user group + +## Search for group + +Get all groups that match your string in their name or path. + +``` +GET /groups?search=foobar +``` + +```json +[ + { + "id": 1, + "name": "Foobar Group", + "path": "foo-bar", + "description": "An interesting group" + } +] +``` ## Group members @@ -124,10 +146,24 @@ POST /groups/:id/members Parameters: -- `id` (required) - The ID of a group +- `id` (required) - The ID or path of a group - `user_id` (required) - The ID of a user to add - `access_level` (required) - Project access level +### Edit group team member + +Updates a group team member to a specified access level. + +``` +PUT /groups/:id/members/:user_id +``` + +Parameters: + +- `id` (required) - The ID of a group +- `user_id` (required) - The ID of a group member +- `access_level` (required) - Project access level + ### Remove user team member Removes user from user team. @@ -138,5 +174,5 @@ DELETE /groups/:id/members/:user_id Parameters: -- `id` (required) - The ID of a user group +- `id` (required) - The ID or path of a user group - `user_id` (required) - The ID of a group member diff --git a/doc/api/issues.md b/doc/api/issues.md index ceeb683a6bf..5a2f6a4c229 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -18,6 +18,8 @@ Parameters: - `state` (optional) - Return `all` issues or just those that are `opened` or `closed` - `labels` (optional) - Comma-separated list of label names +- `order_by` (optional) - Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` +- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` ```json [ @@ -56,7 +58,7 @@ Parameters: "title": "v1.0", "description": "", "due_date": "2012-07-20", - "state": "reopenend", + "state": "reopened", "updated_at": "2012-07-04T13:42:48Z", "created_at": "2012-07-04T13:42:48Z" }, @@ -105,6 +107,8 @@ Parameters: - `state` (optional) - Return `all` issues or just those that are `opened` or `closed` - `labels` (optional) - Comma-separated list of label names - `milestone` (optional) - Milestone title +- `order_by` (optional) - Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` +- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` ## Single issue diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 14884e53915..1f3fd26a241 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -2,7 +2,9 @@ ## List merge requests -Get all merge requests for this project. The `state` parameter can be used to get only merge requests with a given state (`opened`, `closed`, or `merged`) or all of them (`all`). The pagination parameters `page` and `per_page` can be used to restrict the list of merge requests. +Get all merge requests for this project. +The `state` parameter can be used to get only merge requests with a given state (`opened`, `closed`, or `merged`) or all of them (`all`). +The pagination parameters `page` and `per_page` can be used to restrict the list of merge requests. ``` GET /projects/:id/merge_requests @@ -14,8 +16,8 @@ Parameters: - `id` (required) - The ID of a project - `state` (optional) - Return `all` requests or just those that are `merged`, `opened` or `closed` -- `order_by` (optional) - Return requests ordered by `created_at` or `updated_at` fields -- `sort` (optional) - Return requests sorted in `asc` or `desc` order +- `order_by` (optional) - Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` +- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` ```json [ @@ -94,6 +96,76 @@ Parameters: } ``` +## Get single MR changes + +Shows information about the merge request including its files and changes + +``` +GET /projects/:id/merge_request/:merge_request_id/changes +``` + +Parameters: + +- `id` (required) - The ID of a project +- `merge_request_id` (required) - The ID of MR + +```json +{ + "id": 21, + "iid": 1, + "project_id": 4, + "title": "Blanditiis beatae suscipit hic assumenda et molestias nisi asperiores repellat et.", + "description": "Qui voluptatibus placeat ipsa alias quasi. Deleniti rem ut sint. Optio velit qui distinctio.", + "state": "reopened", + "created_at": "2015-02-02T19:49:39.159Z", + "updated_at": "2015-02-02T20:08:49.959Z", + "target_branch": "secret_token", + "source_branch": "version-1-9", + "upvotes": 0, + "downvotes": 0, + "author": { + "name": "Chad Hamill", + "username": "jarrett", + "id": 5, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/b95567800f828948baf5f4160ebb2473?s=40&d=identicon" + }, + "assignee": { + "name": "Administrator", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40&d=identicon" + }, + "source_project_id": 4, + "target_project_id": 4, + "labels": [ ], + "milestone": { + "id": 5, + "iid": 1, + "project_id": 4, + "title": "v2.0", + "description": "Assumenda aut placeat expedita exercitationem labore sunt enim earum.", + "state": "closed", + "created_at": "2015-02-02T19:49:26.013Z", + "updated_at": "2015-02-02T19:49:26.013Z", + "due_date": null + }, + "files": [ + { + "old_path": "VERSION", + "new_path": "VERSION", + "a_mode": "100644", + "b_mode": "100644", + "diff": "--- a/VERSION\ +++ b/VERSION\ @@ -1 +1 @@\ -1.9.7\ +1.9.8", + "new_file": false, + "renamed_file": false, + "deleted_file": false + } + ] +} +``` + ## Create MR Creates a new merge request. @@ -109,6 +181,7 @@ Parameters: - `target_branch` (required) - The target branch - `assignee_id` (optional) - Assignee user ID - `title` (required) - Title of MR +- `description` (optional) - Description of MR - `target_project_id` (optional) - The target project (numeric id) ```json @@ -160,6 +233,7 @@ Parameters: - `target_branch` - The target branch - `assignee_id` - Assignee user ID - `title` - Title of MR +- `description` - Description of MR - `state_event` - New state (close|reopen|merge) ```json @@ -169,6 +243,7 @@ Parameters: "source_branch": "test1", "project_id": 3, "title": "test1", + "description": "description1", "state": "opened", "upvotes": 0, "downvotes": 0, diff --git a/doc/api/milestones.md b/doc/api/milestones.md index 2f525327504..d48b3bcce8a 100644 --- a/doc/api/milestones.md +++ b/doc/api/milestones.md @@ -72,3 +72,16 @@ Parameters: - `description` (optional) - The description of a milestone - `due_date` (optional) - The due date of the milestone - `state_event` (optional) - The state event of the milestone (close|activate) + +## Get all issues assigned to a single milestone + +Gets all issues assigned to a single project milestone. + +``` +GET /projects/:id/milestones/:milestone_id/issues +``` + +Parameters: + +- `id` (required) - The ID of a project +- `milestone_id` (required) - The ID of a project milestone diff --git a/doc/api/notes.md b/doc/api/notes.md index b5256ac803e..c22e493562a 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -78,6 +78,21 @@ Parameters: - `issue_id` (required) - The ID of an issue - `body` (required) - The content of a note +### Modify existing issue note + +Modify existing note of an issue. + +``` +PUT /projects/:id/issues/:issue_id/notes/:note_id +``` + +Parameters: + +- `id` (required) - The ID of a project +- `issue_id` (required) - The ID of an issue +- `note_id` (required) - The ID of a note +- `body` (required) - The content of a note + ## Snippets ### List all snippet notes @@ -137,7 +152,22 @@ POST /projects/:id/snippets/:snippet_id/notes Parameters: - `id` (required) - The ID of a project -- `snippet_id` (required) - The ID of an snippet +- `snippet_id` (required) - The ID of a snippet +- `body` (required) - The content of a note + +### Modify existing snippet note + +Modify existing note of a snippet. + +``` +PUT /projects/:id/snippets/:snippet_id/notes/:note_id +``` + +Parameters: + +- `id` (required) - The ID of a project +- `snippet_id` (required) - The ID of a snippet +- `note_id` (required) - The ID of a note - `body` (required) - The content of a note ## Merge Requests @@ -199,3 +229,18 @@ Parameters: - `id` (required) - The ID of a project - `merge_request_id` (required) - The ID of a merge request - `body` (required) - The content of a note + +### Modify existing merge request note + +Modify existing note of a merge request. + +``` +PUT /projects/:id/merge_requests/:merge_request_id/notes/:note_id +``` + +Parameters: + +- `id` (required) - The ID of a project +- `merge_request_id` (required) - The ID of a merge request +- `note_id` (required) - The ID of a note +- `body` (required) - The content of a note diff --git a/doc/api/oauth2.md b/doc/api/oauth2.md new file mode 100644 index 00000000000..d416a826f79 --- /dev/null +++ b/doc/api/oauth2.md @@ -0,0 +1,102 @@ +# GitLab as an OAuth2 client + +This document is about using other OAuth authentication service providers to sign into GitLab. +If you want GitLab to be an OAuth authentication service provider to sign into other services please see the [Oauth2 provider documentation](../integration/oauth_provider.md). + +OAuth2 is a protocol that enables us to authenticate a user without requiring them to give their password. + +Before using the OAuth2 you should create an application in user's account. Each application gets a unique App ID and App Secret parameters. You should not share these. + +This functionality is based on [doorkeeper gem](https://github.com/doorkeeper-gem/doorkeeper) + +## Web Application Flow + +This flow is using for authentication from third-party web sites and is probably used the most. +It basically consists of an exchange of an authorization token for an access token. For more detailed info, check out the [RFC spec here](http://tools.ietf.org/html/rfc6749#section-4.1) + +This flow consists from 3 steps. + +### 1. Registering the client + +Create an application in user's account profile. + +### 2. Requesting authorization + +To request the authorization token, you should visit the `/oauth/authorize` endpoint. You can do that by visiting manually the URL: + +``` +http://localhost:3000/oauth/authorize?client_id=APP_ID&redirect_uri=REDIRECT_URI&response_type=code +``` + +Where REDIRECT_URI is the URL in your app where users will be sent after authorization. + +### 3. Requesting the access token + +To request the access token, you should use the returned code and exchange it for an access token. To do that you can use any HTTP client. In this case, I used rest-client: + +``` +parameters = 'client_id=APP_ID&client_secret=APP_SECRET&code=RETURNED_CODE&grant_type=AUTHORIZATION_CODE&redirect_uri=REDIRECT_URI' +RestClient.post 'http://localhost:3000/oauth/token', parameters + +# The response will be +{ + "access_token": "de6780bc506a0446309bd9362820ba8aed28aa506c71eedbe1c5c4f9dd350e54", + "token_type": "bearer", + "expires_in": 7200, + "refresh_token": "8257e65c97202ed1726cf9571600918f3bffb2544b26e00a61df9897668c33a1" +} +``` + +You can now make requests to the API with the access token returned. + +### Use the access token to access the API + +The access token allows you to make requests to the API on a behalf of a user. + +``` +GET https://localhost:3000/api/v3/user?access_token=OAUTH-TOKEN +``` + +Or you can put the token to the Authorization header: + +``` +curl -H "Authorization: Bearer OAUTH-TOKEN" https://localhost:3000/api/v3/user +``` + +## Resource Owner Password Credentials + +In this flow, a token is requested in exchange for the resource owner credentials (username and password). +The credentials should only be used when there is a high degree of trust between the resource owner and the client (e.g. the +client is part of the device operating system or a highly privileged application), and when other authorization grant types are not +available (such as an authorization code). + +Even though this grant type requires direct client access to the resource owner credentials, the resource owner credentials are used +for a single request and are exchanged for an access token. This grant type can eliminate the need for the client to store the +resource owner credentials for future use, by exchanging the credentials with a long-lived access token or refresh token. +You can do POST request to `/oauth/token` with parameters: + +``` +{ + "grant_type" : "password", + "username" : "user@example.com", + "password" : "sekret" +} +``` + +Then, you'll receive the access token back in the response: + +``` +{ + "access_token": "1f0af717251950dbd4d73154fdf0a474a5c5119adad999683f5b450c460726aa", + "token_type": "bearer", + "expires_in": 7200 +} +``` + +For testing you can use the oauth2 ruby gem: + +``` +client = OAuth2::Client.new('the_client_id', 'the_client_secret', :site => "http://example.com") +access_token = client.password.get_token('user@example.com', 'sekret') +puts access_token.token +``` diff --git a/doc/api/projects.md b/doc/api/projects.md index 0055e2e476f..a1a23051d7e 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -1,5 +1,23 @@ # Projects + +### Project visibility level + +Project in GitLab has be either private, internal or public. +You can determine it by `visibility_level` field in project. + +Constants for project visibility levels are next: + +* Private. `visibility_level` is `0`. + Project access must be granted explicitly for each user. + +* Internal. `visibility_level` is `10`. + The project can be cloned by any logged in user. + +* Public. `visibility_level` is `20`. + The project can be cloned without any authentication. + + ## List projects Get a list of projects accessible by the authenticated user. @@ -11,6 +29,9 @@ GET /projects Parameters: - `archived` (optional) - if passed, limit by archived status +- `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at` +- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` +- `search` (optional) - Return list of authorized projects according to a search criteria ```json [ @@ -95,6 +116,13 @@ Get a list of projects which are owned by the authenticated user. GET /projects/owned ``` +Parameters: + +- `archived` (optional) - if passed, limit by archived status +- `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at` +- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` +- `search` (optional) - Return list of authorized projects according to a search criteria + ### List ALL projects Get a list of all GitLab projects (admin only). @@ -103,6 +131,13 @@ Get a list of all GitLab projects (admin only). GET /projects/all ``` +Parameters: + +- `archived` (optional) - if passed, limit by archived status +- `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at` +- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` +- `search` (optional) - Return list of authorized projects according to a search criteria + ### Get single project Get a specific project, identified by project ID or NAMESPACE/PROJECT_NAME, which is owned by the authenticated user. @@ -284,6 +319,31 @@ Parameters: - `visibility_level` (optional) - `import_url` (optional) +### Edit project + +Updates an existing project + +``` +PUT /projects/:id +``` + +Parameters: + +- `id` (required) - The ID of a project +- `name` (optional) - project name +- `path` (optional) - repository name for project +- `description` (optional) - short project description +- `default_branch` (optional) +- `issues_enabled` (optional) +- `merge_requests_enabled` (optional) +- `wiki_enabled` (optional) +- `snippets_enabled` (optional) +- `public` (optional) - if `true` same as setting visibility_level = 20 +- `visibility_level` (optional) + +On success, method returns 200 with the updated project. If parameters are +invalid, 400 is returned. + ### Fork project Forks a project into the user namespace of the authenticated user. @@ -513,7 +573,7 @@ Parameters: } ], "tree": "c68537c6534a02cc2b176ca1549f4ffa190b58ee", - "message": "give caolan credit where it's due (up top)", + "message": "give Caolan credit where it's due (up top)", "author": { "name": "Jeremy Ashkenas", "email": "jashkenas@example.com" @@ -628,6 +688,8 @@ GET /projects/search/:query Parameters: -- query (required) - A string contained in the project name -- per_page (optional) - number of projects to return per page -- page (optional) - the page to retrieve +- `query` (required) - A string contained in the project name +- `per_page` (optional) - number of projects to return per page +- `page` (optional) - the page to retrieve +- `order_by` (optional) - Return requests ordered by `id`, `name`, `created_at` or `last_activity_at` fields +- `sort` (optional) - Return requests sorted in `asc` or `desc` order diff --git a/doc/api/services.md b/doc/api/services.md index ab9f9c00c67..cbf767d1b25 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -13,7 +13,7 @@ PUT /projects/:id/services/gitlab-ci Parameters: - `token` (required) - CI project token -- `project_url` (required) - CI project url +- `project_url` (required) - CI project URL ### Delete GitLab CI service @@ -23,23 +23,23 @@ Delete GitLab CI service settings for a project. DELETE /projects/:id/services/gitlab-ci ``` -## Hipchat +## HipChat -### Edit Hipchat service +### Edit HipChat service -Set Hipchat service for project. +Set HipChat service for project. ``` PUT /projects/:id/services/hipchat ``` Parameters: -- `token` (required) - Hipchat token -- `room` (required) - Hipchat room name +- `token` (required) - HipChat token +- `room` (required) - HipChat room name -### Delete Hipchat service +### Delete HipChat service -Delete Hipchat service for a project. +Delete HipChat service for a project. ``` DELETE /projects/:id/services/hipchat diff --git a/doc/api/users.md b/doc/api/users.md index fec5deebee4..a8b7685b503 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -78,7 +78,8 @@ GET /users "is_admin": false, "avatar_url": "http://localhost:3000/uploads/user/avatar/1/cd8.jpeg", "can_create_group": true, - "can_create_project": true + "can_create_project": true, + "projects_limit": 100 } ] ``` @@ -140,7 +141,8 @@ Parameters: "color_scheme_id": 2, "is_admin": false, "can_create_group": true, - "can_create_project": true + "can_create_project": true, + "projects_limit": 100 } ``` @@ -241,7 +243,8 @@ GET /user "color_scheme_id": 2, "is_admin": false, "can_create_group": true, - "can_create_project": true + "can_create_project": true, + "projects_limit": 100 } ``` @@ -258,12 +261,14 @@ GET /user/keys { "id": 1, "title": "Public key", - "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=" + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=", + "created_at": "2014-08-01T14:47:39.080Z" }, { "id": 3, "title": "Another Public key", - "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=" + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=", + "created_at": "2014-08-01T14:47:39.080Z" } ] ``` @@ -300,7 +305,8 @@ Parameters: { "id": 1, "title": "Public key", - "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=" + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=", + "created_at": "2014-08-01T14:47:39.080Z" } ``` @@ -317,6 +323,31 @@ Parameters: - `title` (required) - new SSH Key's title - `key` (required) - new SSH key +```json +{ + "created_at": "2015-01-21T17:44:33.512Z", + "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAMLrhYgI3atfrSD6KDas1b/3n6R/HP+bLaHHX6oh+L1vg31mdUqK0Ac/NjZoQunavoyzqdPYhFz9zzOezCrZKjuJDS3NRK9rspvjgM0xYR4d47oNZbdZbwkI4cTv/gcMlquRy0OvpfIvJtjtaJWMwTLtM5VhRusRuUlpH99UUVeXAAAAFQCVyX+92hBEjInEKL0v13c/egDCTQAAAIEAvFdWGq0ccOPbw4f/F8LpZqvWDydAcpXHV3thwb7WkFfppvm4SZte0zds1FJ+Hr8Xzzc5zMHe6J4Nlay/rP4ewmIW7iFKNBEYb/yWa+ceLrs+TfR672TaAgO6o7iSRofEq5YLdwgrwkMmIawa21FrZ2D9SPao/IwvENzk/xcHu7YAAACAQFXQH6HQnxOrw4dqf0NqeKy1tfIPxYYUZhPJfo9O0AmBW2S36pD2l14kS89fvz6Y1g8gN/FwFnRncMzlLY/hX70FSc/3hKBSbH6C6j8hwlgFKfizav21eS358JJz93leOakJZnGb8XlWvz1UJbwCsnR2VEY8Dz90uIk1l/UqHkA= loic@call", + "title": "ABC", + "id": 4 +} +``` + +Will return created key with status `201 Created` on success. If an +error occurs a `400 Bad Request` is returned with a message explaining the error: + +```json +{ + "message": { + "fingerprint": [ + "has already been taken" + ], + "key": [ + "has already been taken" + ] + } +} +``` + ## Add SSH key for user Create new key owned by specified user. Available only for admin diff --git a/doc/customization/libravatar.md b/doc/customization/libravatar.md index 4dffd3027a9..ee57fdc6590 100644 --- a/doc/customization/libravatar.md +++ b/doc/customization/libravatar.md @@ -16,7 +16,7 @@ the configuration options as follows: ```yml gravatar: enabled: true - # gravatar urls: possible placeholders: %{hash} %{size} %{email} + # gravatar URLs: possible placeholders: %{hash} %{size} %{email} plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" ``` @@ -25,14 +25,14 @@ the configuration options as follows: ```yml gravatar: enabled: true - # gravatar urls: possible placeholders: %{hash} %{size} %{email} + # gravatar URLs: possible placeholders: %{hash} %{size} %{email} ssl_url: "https://seccdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" ``` ## Self-hosted -If you are [running your own libravatar service](http://wiki.libravatar.org/running_your_own/) the url will be different in the configuration -but the important part is to provide the same placeholders so GitLab can parse the url correctly. +If you are [running your own libravatar service](http://wiki.libravatar.org/running_your_own/) the URL will be different in the configuration +but the important part is to provide the same placeholders so GitLab can parse the URL correctly. For example, you host a service on `http://libravatar.example.com` the `plain_url` you need to supply in `gitlab.yml` is @@ -65,5 +65,5 @@ Run `sudo gitlab-ctl reconfigure` for changes to take effect. [Libravatar supports different sets](http://wiki.libravatar.org/api/) of `missing images` for emails not found on the Libravatar service. -In order to use a different set other than `identicon`, replace `&d=identicon` portion of the url with another supported set. -For example, you can use `retro` set in which case url would look like: `plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=retro"` +In order to use a different set other than `identicon`, replace `&d=identicon` portion of the URL with another supported set. +For example, you can use `retro` set in which case the URL would look like: `plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=retro"` diff --git a/doc/development/README.md b/doc/development/README.md index 20db6662aca..c31e5d7ae97 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -4,3 +4,4 @@ - [Shell commands](shell_commands.md) in the GitLab codebase - [Rake tasks](rake_tasks.md) for development - [CI setup](ci_setup.md) for testing GitLab +- [Sidekiq debugging](sidekiq_debugging.md) diff --git a/doc/development/architecture.md b/doc/development/architecture.md index c4813d22eaa..714cc016004 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -8,6 +8,38 @@ EE releases are available not long after CE releases. To obtain the GitLab EE th Both EE and CE require an add-on component called gitlab-shell. It is obtained from the [gitlab-shell repository](https://gitlab.com/gitlab-org/gitlab-shell/tree/master). New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical. +## Physical office analogy + +You can imagine GitLab as a physical office. + +**The repositories** are the goods GitLab handling. +They can be stored in a warehouse. +This can be either a hard disk, or something more complex, such as a NFS filesystem; + +**Nginx** acts like the front-desk. +Users come to Nginx and request actions to be done by workers in the office; + +**The database** is a series of metal file cabinets with information on: + - The goods in the warehouse (metadata, issues, merge requests etc); + - The users coming to the front desk (permissions) + +**Redis** is a communication board with “cubby holes” that can contain tasks for office workers; + +**Sidekiq** is a worker that primarily handles sending out emails. +It takes tasks from the Redis communication board; + +**A Unicorn worker** is a worker that handles quick/mundane tasks. +They work with the communication board (Redis). +Their job description: + - check permissions by checking the user session stored in a Redis “cubby hole”; + - make tasks for Sidekiq; + - fetch stuff from the warehouse or move things around in there; + +**Gitlab-shell** is a third kind of worker that takes orders from a fax machine (SSH) instead of the front desk (HTTP). +Gitlab-shell communicates with Sidekiq via the “communication board” (Redis), and asks quick questions of the Unicorn workers either directly or via the front desk. + +**GitLab Enterprise Edition (the application)** is the collection of processes and business practices that the office is run by. + ## System Layout When referring to ~git in the pictures it means the home directory of the git user which is typically /home/git. @@ -38,7 +70,7 @@ To summarize here's the [directory structure of the `git` user home directory](. ps aux | grep '^git' -GitLab has several components to operate. As a system user (i.e. any user that is not the `git` user) it requires a persistent database (MySQL/PostreSQL) and redis database. It also uses Apache httpd or nginx to proxypass Unicorn. As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server running on port `8080` by default). Under the GitLab user there are normally 4 processes: `unicorn_rails master` (1 process), `unicorn_rails worker` (2 processes), `sidekiq` (1 process). +GitLab has several components to operate. As a system user (i.e. any user that is not the `git` user) it requires a persistent database (MySQL/PostreSQL) and redis database. It also uses Apache httpd or Nginx to proxypass Unicorn. As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server running on port `8080` by default). Under the GitLab user there are normally 4 processes: `unicorn_rails master` (1 process), `unicorn_rails worker` (2 processes), `sidekiq` (1 process). ### Repository access @@ -114,13 +146,13 @@ nginx Apache httpd -- [Explanation of apache logs](http://httpd.apache.org/docs/2.2/logs.html). +- [Explanation of Apache logs](http://httpd.apache.org/docs/2.2/logs.html). - `/var/log/apache2/` contains error and output logs (on Ubuntu). - `/var/log/httpd/` contains error and output logs (on RHEL). redis -- `/var/log/redis/redis.log` there are also logrotated logs there. +- `/var/log/redis/redis.log` there are also log-rotated logs there. PostgreSQL diff --git a/doc/development/ci_setup.md b/doc/development/ci_setup.md index ee16aedafe7..f417667754e 100644 --- a/doc/development/ci_setup.md +++ b/doc/development/ci_setup.md @@ -26,7 +26,7 @@ We use [these build scripts](https://gitlab.com/gitlab-org/gitlab-ci/blob/master # Build configuration on [Semaphore](https://semaphoreapp.com/gitlabhq/gitlabhq/) for testing the [GitHub.com repo](https://github.com/gitlabhq/gitlabhq) - Language: Ruby -- Ruby verion: 2.1.2 +- Ruby version: 2.1.2 - database.yml: pg Build commands diff --git a/doc/development/omnibus.md b/doc/development/omnibus.md new file mode 100644 index 00000000000..0ba354d28a2 --- /dev/null +++ b/doc/development/omnibus.md @@ -0,0 +1,32 @@ +# What you should know about omnibus packages + +Most users install GitLab using our omnibus packages. As a developer it can be +good to know how the omnibus packages differ from what you have on your laptop +when you are coding. + +## Files are owned by root by default + +All the files in the Rails tree (`app/`, `config/` etc.) are owned by 'root' in +omnibus installations. This makes the installation simpler and it provides +extra security. The omnibus reconfigure script contains commands that give +write access to the 'git' user only where needed. + +For example, the 'git' user is allowed to write in the `log/` directory, in +`public/uploads`, and they are allowed to rewrite the `db/schema.rb` file. + +In other cases, the reconfigure script tricks GitLab into not trying to write a +file. For instance, GitLab will generate a `.secret` file if it cannot find one +and write it to the Rails root. In the omnibus packages, reconfigure writes the +`.secret` file first, so that GitLab never tries to write it. + +## Code, data and logs are in separate directories + +The omnibus design separates code (read-only, under `/opt/gitlab`) from data +(read/write, under `/var/opt/gitlab`) and logs (read/write, under +`/var/log/gitlab`). To make this happen the reconfigure script sets custom +paths where it can in GitLab config files, and where there are no path +settings, it uses symlinks. + +For example, `config/gitlab.yml` is treated as data so that file is a symlink. +The same goes for `public/uploads`. The `log/` directory is replaced by omnibus +with a symlink to `/var/log/gitlab/gitlab-rails`. diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 6d9ac161e91..53f8095cb13 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -1,6 +1,6 @@ # Rake tasks for developers -## Setup db with developer seeds: +## Setup db with developer seeds Note that if your db user does not have advanced privileges you must create the db manually before running this command. @@ -8,6 +8,10 @@ Note that if your db user does not have advanced privileges you must create the bundle exec rake setup ``` +The `setup` task is a alias for `gitlab:setup`. +This tasks calls `db:setup` to create the database, calls `add_limits_mysql` that adds limits to the database schema in case of a MySQL database and fianlly it calls `db:seed_fu` to seed the database. +Note: `db:setup` calls `db:seed` but this does nothing. + ## Run tests This runs all test suites present in GitLab. diff --git a/doc/development/shell_commands.md b/doc/development/shell_commands.md index 23c8365c340..42f17e19536 100644 --- a/doc/development/shell_commands.md +++ b/doc/development/shell_commands.md @@ -1,5 +1,8 @@ # Guidelines for shell commands in the GitLab codebase +This document contains guidelines for working with processes and files in the GitLab codebase. +These guidelines are meant to make your code more reliable _and_ secure. + ## References - [Google Ruby Security Reviewer's Guide](https://code.google.com/p/ruby-security/wiki/Guide) @@ -105,7 +108,67 @@ In other repositories, such as gitlab-shell you can also use `IO.popen`. ```ruby # Safe IO.popen example -logs = IO.popen(%W(git log), chdir: repo_dir).read +logs = IO.popen(%W(git log), chdir: repo_dir) { |p| p.read } ``` Note that unlike `Gitlab::Popen.popen`, `IO.popen` does not capture standard error. + +## Avoid user input at the start of path strings + +Various methods for opening and reading files in Ruby can be used to read the +standard output of a process instead of a file. The following two commands do +roughly the same: + +``` +`touch /tmp/pawned-by-backticks` +File.read('|touch /tmp/pawned-by-file-read') +``` + +The key is to open a 'file' whose name starts with a `|`. +Affected methods include Kernel#open, File::read, File::open, IO::open and IO::read. + +You can protect against this behavior of 'open' and 'read' by ensuring that an +attacker cannot control the start of the filename string you are opening. For +instance, the following is sufficient to protect against accidentally starting +a shell command with `|`: + +``` +# we assume repo_path is not controlled by the attacker (user) +path = File.join(repo_path, user_input) +# path cannot start with '|' now. +File.read(path) +``` + +## Guard against path traversal + +Path traversal is a security where the program (GitLab) tries to restrict user +access to a certain directory on disk, but the user manages to open a file +outside that directory by taking advantage of the `../` path notation. + +``` +# Suppose the user gave us a path and they are trying to trick us +user_input = '../other-repo.git/other-file' + +# We look up the repo path somewhere +repo_path = 'repositories/user-repo.git' + +# The intention of the code below is to open a file under repo_path, but +# because the user used '..' she can 'break out' into +# 'repositories/other-repo.git' +full_path = File.join(repo_path, user_input) +File.open(full_path) do # Oops! +``` + +A good way to protect against this is to compare the full path with its +'absolute path' according to Ruby's `File.absolute_path`. + +``` +full_path = File.join(repo_path, user_input) +if full_path != File.absolute_path(full_path) + raise "Invalid path: #{full_path.inspect}" +end + +File.open(full_path) do # Etc. +``` + +A check like this could have avoided CVE-2013-4583. diff --git a/doc/development/sidekiq_debugging.md b/doc/development/sidekiq_debugging.md new file mode 100644 index 00000000000..cea11e5f126 --- /dev/null +++ b/doc/development/sidekiq_debugging.md @@ -0,0 +1,14 @@ +# Sidekiq debugging + +## Log arguments to Sidekiq jobs + +If you want to see what arguments are being passed to Sidekiq jobs you can set +the SIDEKIQ_LOG_ARGUMENTS environment variable. + +``` +SIDEKIQ_LOG_ARGUMENTS=1 bundle exec foreman start +``` + +It is not recommend to enable this setting in production because some Sidekiq +jobs (such as sending a password reset email) take secret arguments (for +example the password reset token). diff --git a/doc/hooks/custom_hooks.md b/doc/hooks/custom_hooks.md new file mode 100644 index 00000000000..f7d4f3de68b --- /dev/null +++ b/doc/hooks/custom_hooks.md @@ -0,0 +1,41 @@ +# Custom Git Hooks + +**Note: Custom git hooks must be configured on the filesystem of the GitLab +server. Only GitLab server administrators will be able to complete these tasks. +Please explore webhooks as an option if you do not have filesystem access.** + +Git natively supports hooks that are executed on different actions. +Examples of server-side git hooks include pre-receive, post-receive, and update. +See +[Git SCM Server-Side Hooks](http://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#Server-Side-Hooks) +for more information about each hook type. + +As of gitlab-shell version 2.2.0 (which requires GitLab 7.5+), GitLab +administrators can add custom git hooks to any GitLab project. + +## Setup + +Normally, git hooks are placed in the repository or project's `hooks` directory. +GitLab creates a symlink from each project's `hooks` directory to the +gitlab-shell `hooks` directory for ease of maintenance between gitlab-shell +upgrades. As such, custom hooks are implemented a little differently. Behavior +is exactly the same once the hook is created, though. Follow these steps to +set up a custom hook. + +1. Pick a project that needs a custom git hook. +1. On the GitLab server, navigate to the project's repository directory. +For an installation from source the path is usually +`/home/git/repositories/<group>/<project>.git`. For Omnibus installs the path is +usually `/var/opt/gitlab/git-data/repositories/<group>/<project>.git`. +1. Create a new directory in this location called `custom_hooks`. +1. Inside the new `custom_hooks` directory, create a file with a name matching +the hook type. For a pre-receive hook the file name should be `pre-receive` with +no extension. +1. Make the hook file executable and make sure it's owned by git. +1. Write the code to make the git hook function as expected. Hooks can be +in any language. Ensure the 'shebang' at the top properly reflects the language +type. For example, if the script is in Ruby the shebang will probably be +`#!/usr/bin/env ruby`. + +That's it! Assuming the hook code is properly implemented the hook will fire +as appropriate. diff --git a/doc/install/installation.md b/doc/install/installation.md index b5f54ff9b28..28597fd39d2 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -2,13 +2,13 @@ ## Consider the Omnibus package installation -Since a manual installation is a lot of work and error prone we strongly recommend the fast and reliable [Omnibus package installation](https://about.gitlab.com/downloads/) (deb/rpm). +Since an installation from source is a lot of work and error prone we strongly recommend the fast and reliable [Omnibus package installation](https://about.gitlab.com/downloads/) (deb/rpm). ## Select Version to Install -Make sure you view [this installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md) from the branch (version) of GitLab you would like to install. In most cases this should be the highest numbered stable branch (example shown below). - - +Make sure you view [this installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md) from the tag (version) of GitLab you would like to install. +In most cases this should be the highest numbered production tag (without rc in it). +You can select the tag in the version dropdown in the top left corner of GitLab (below the menu bar). If the highest number stable branch is unclear please check the [GitLab Blog](https://about.gitlab.com/blog/) for installation guide links by version. @@ -22,7 +22,9 @@ This is the official installation guide to set up a production server. To set up The following steps have been known to work. Please **use caution when you deviate** from this guide. Make sure you don't violate any assumptions GitLab makes about its environment. For example many people run into permission problems because they changed the location of directories or run services as the wrong user. -If you find a bug/error in this guide please **submit a merge request** following the [contributing guide](../../CONTRIBUTING.md). +If you find a bug/error in this guide please **submit a merge request** +following the +[contributing guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md). ## Overview @@ -54,7 +56,7 @@ up-to-date and install it. Install the required packages (needed to compile Ruby and native extensions to Ruby gems): - sudo apt-get install -y build-essential zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libreadline-dev libncurses5-dev libffi-dev curl openssh-server redis-server checkinstall libxml2-dev libxslt-dev libcurl4-openssl-dev libicu-dev logrotate python-docutils pkg-config cmake + sudo apt-get install -y build-essential zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libreadline-dev libncurses5-dev libffi-dev curl openssh-server redis-server checkinstall libxml2-dev libxslt-dev libcurl4-openssl-dev libicu-dev logrotate python-docutils pkg-config cmake libkrb5-dev Make sure you have the right version of Git installed @@ -101,8 +103,8 @@ Remove the old Ruby 1.8 if present Download Ruby and compile it: mkdir /tmp/ruby && cd /tmp/ruby - curl -L --progress ftp://ftp.ruby-lang.org/pub/ruby/2.1/ruby-2.1.2.tar.gz | tar xz - cd ruby-2.1.2 + curl -L --progress http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.5.tar.gz | tar xz + cd ruby-2.1.5 ./configure --disable-install-rdoc make sudo make install @@ -127,7 +129,8 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da # Login to PostgreSQL sudo -u postgres psql -d template1 - # Create a user for GitLab. + # Create a user for GitLab + # Do not type the 'template1=#', this is part of the prompt template1=# CREATE USER git CREATEDB; # Create the GitLab production database & grant all privileges on database @@ -139,6 +142,9 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da # Try connecting to the new database with the new user sudo -u git -H psql -d gitlabhq_production + # Quit the database session + gitlabhq_production> \q + ## 5. Redis sudo apt-get install redis-server @@ -177,9 +183,9 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da ### Clone the Source # Clone GitLab repository - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 7-4-stable gitlab + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 7-8-stable gitlab -**Note:** You can change `7-4-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! +**Note:** You can change `7-8-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! ### Configure It @@ -195,7 +201,7 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da # Make sure GitLab can write to the log/ and tmp/ directories sudo chown -R git log/ sudo chown -R git tmp/ - sudo chmod -R u+rwX log/ + sudo chmod -R u+rwX,go-w log/ sudo chmod -R u+rwX tmp/ # Create directory for satellites @@ -274,7 +280,7 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da GitLab Shell is an SSH access and repository management software developed specially for GitLab. # Run the installation task for gitlab-shell (replace `REDIS_URL` if needed): - sudo -u git -H bundle exec rake gitlab:shell:install[v2.0.1] REDIS_URL=unix:/var/run/redis/redis.sock RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:shell:install[v2.5.4] REDIS_URL=unix:/var/run/redis/redis.sock RAILS_ENV=production # By default, the gitlab-shell config is generated from your main GitLab config. # You can review (and modify) the gitlab-shell config as follows: @@ -290,9 +296,9 @@ GitLab Shell is an SSH access and repository management software developed speci # When done you see 'Administrator account created:' -**Note:** You can set the Administrator password by supplying it in environmental variable `GITLAB_ROOT_PASSWORD`, eg.: +**Note:** You can set the Administrator/root password by supplying it in environmental variable `GITLAB_ROOT_PASSWORD` as seen below. If you don't set the password (and it is set to the default one) please wait with exposing GitLab to the public internet until the installation is done and you've logged into the server the first time. During the first login you'll be forced to change the default password. - sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production GITLAB_ROOT_PASSWORD=newpassword + sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production GITLAB_ROOT_PASSWORD=yourpassword ### Install Init Script @@ -379,15 +385,17 @@ NOTE: Supply `SANITIZE=true` environment variable to `gitlab:check` to omit proj ### Initial Login -Visit YOUR_SERVER in your web browser for your first GitLab login. The setup has created an admin account for you. You can use it to log in: +Visit YOUR_SERVER in your web browser for your first GitLab login. The setup has created a default admin account for you. You can use it to log in: root 5iveL!fe -**Important Note:** Please go over to your profile page and immediately change the password, so nobody can access your GitLab by using this login information later on. +**Important Note:** On login you'll be prompted to change the password. **Enjoy!** +You can use `sudo service gitlab start` and `sudo service gitlab stop` to start and stop GitLab. + ## Advanced Setup Tips ### Using HTTPS @@ -451,4 +459,4 @@ You can configure LDAP authentication in `config/gitlab.yml`. Please restart Git ### Using Custom Omniauth Providers -See the [omniauth integration document](doc/integration/omniauth.md) +See the [omniauth integration document](../integration/omniauth.md) diff --git a/doc/install/requirements.md b/doc/install/requirements.md index ed194253148..5bdb9caa2bf 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -7,7 +7,7 @@ - Ubuntu - Debian - CentOS -- RedHat Enterprise Linux (please use the CentOS packages and instructions) +- Red Hat Enterprise Linux (please use the CentOS packages and instructions) - Scientific Linux (please use the CentOS packages and instructions) - Oracle Linux (please use the CentOS packages and instructions) @@ -22,7 +22,7 @@ For the installations options please see [the installation page on the GitLab we - FreeBSD On the above unsupported distributions is still possible to install GitLab yourself. -Please see the [manual installation guide](https://github.com/gitlabhq/gitlabhq/blob/master/doc/install/installation.md) and the [unofficial installation guides](https://github.com/gitlabhq/gitlab-public-wiki/wiki/Unofficial-Installation-Guides) on the public wiki for more information. +Please see the [installation from source guide](https://github.com/gitlabhq/gitlabhq/blob/master/doc/install/installation.md) and the [unofficial installation guides](https://github.com/gitlabhq/gitlab-public-wiki/wiki/Unofficial-Installation-Guides) on the public wiki for more information. ### Non-Unix operating systems such as Windows @@ -38,6 +38,16 @@ We love [JRuby](http://jruby.org/) and [Rubinius](http://rubini.us/) but GitLab ## Hardware requirements +### Storage + +The necessary hard drive space largely depends on the size of the repos you want to store in GitLab but as a *rule of thumb* you should have at least twice as much free space as all your repos combined take up. You need twice the storage because [GitLab satellites](structure.md) contain an extra copy of each repo. + +If you want to be flexible about growing your hard drive space in the future consider mounting it using LVM so you can add more hard drives when you need them. + +Apart from a local hard drive you can also mount a volume that supports the network file system (NFS) protocol. This volume might be located on a file server, a network attached storage (NAS) device, a storage area network (SAN) or on an Amazon Web Services (AWS) Elastic Block Store (EBS) volume. + +If you have enough RAM memory and a recent CPU the speed of GitLab is mainly limited by hard drive seek times. Having a fast drive (7200 RPM and up) or a solid state drive (SSD) will improve the responsiveness of GitLab. + ### CPU - 1 core works supports up to 100 users but the application can be a bit slower due to having all workers and background jobs running on the same core @@ -50,11 +60,10 @@ We love [JRuby](http://jruby.org/) and [Rubinius](http://rubini.us/) but GitLab ### Memory -- 512MB is the absolute minimum but we do not recommend this amount of memory. -You will either need to configure 512MB or 1.5GB of swap space. -With 512MB of swap space you must configure only one unicorn worker. -With one unicorn worker only git over ssh access will work because the git over HTTP access requires two running workers (one worker to receive the user request and one worker for the authorization check). -If you use SSD storage and configure 1.5GB of swap space you can use two Unicorn workers, this will allow HTTP access but it will still be slow. +You need at least 2GB of addressable memory (RAM + swap) to install and use GitLab! +With less memory GitLab will give strange errors during the reconfigure run and 500 errors during usage. + +- 512MB RAM + 1.5GB of swap is the absolute minimum but we strongly **advise against** this amount of memory. See the unicorn worker section below for more advise. - 1GB RAM + 1GB swap supports up to 100 users - **2GB RAM** is the **recommended** memory size and supports up to 500 users - 4GB RAM supports up to 2,000 users @@ -65,15 +74,16 @@ If you use SSD storage and configure 1.5GB of swap space you can use two Unicorn Notice: The 25 workers of Sidekiq will show up as separate processes in your process overview (such as top or htop) but they share the same RAM allocation since Sidekiq is a multithreaded application. -### Storage +## Unicorn Workers -The necessary hard drive space largely depends on the size of the repos you want to store in GitLab but as a *rule of thumb* you should have at least twice as much free space as all your repos combined take up. You need twice the storage because [GitLab satellites](structure.md) contain an extra copy of each repo. +It's possible to increase the amount of unicorn workers and tis will usually help for to reduce the response time of the applications. +For most instances we recommend using: CPU cores + 1 = unicorn workers. +So for a machine with 2 cores, 3 unicorn workers is ideal. -If you want to be flexible about growing your hard drive space in the future consider mounting it using LVM so you can add more hard drives when you need them. - -Apart from a local hard drive you can also mount a volume that supports the network file system (NFS) protocol. This volume might be located on a file server, a network attached storage (NAS) device, a storage area network (SAN) or on an Amazon Web Services (AWS) Elastic Block Store (EBS) volume. - -If you have enough RAM memory and a recent CPU the speed of GitLab is mainly limited by hard drive seek times. Having a fast drive (7200 RPM and up) or a solid state drive (SSD) will improve the responsiveness of GitLab. +For all machines that have 1GB and up we recommend a minimum of two unicorn workers. +If you have a 512MB machine with a magnetic (non-SSD) swap drive we recommend to configure only one Unicorn worker to prevent excessive swapping. +With one Unicorn worker only git over ssh access will work because the git over HTTP access requires two running workers (one worker to receive the user request and one worker for the authorization check). +If you have a 512MB machine with a SSD drive you can use two Unicorn workers, this will allow HTTP access although it will be slow due to swapping. ## Database @@ -90,7 +100,7 @@ On a very active server (10,000 active users) the Sidekiq process can use 1GB+ o ## Supported web browsers - Chrome (Latest stable version) -- Firefox (Latest released version and [latest ESR version](https://www.mozilla.org/en-US/firefox/organizations/)) +- Firefox (Latest released version and [latest ESR version](https://www.mozilla.org/en-US/firefox/organizations/)) - Safari 7+ (known problem: required fields in html5 do not work) - Opera (Latest released version) - IE 10+ diff --git a/doc/integration/README.md b/doc/integration/README.md index 357ed038314..e5f33d8deed 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -6,14 +6,14 @@ See the documentation below for details on how to configure these services. - [External issue tracker](external-issue-tracker.md) Redmine, JIRA, etc. - [LDAP](ldap.md) Set up sign in via LDAP -- [OmniAuth](omniauth.md) Sign in via Twitter, GitHub, and Google via OAuth. +- [OmniAuth](omniauth.md) Sign in via Twitter, GitHub, GitLab, and Google via OAuth. - [Slack](slack.md) Integrate with the Slack chat service -Jenkins support is [available in GitLab EE](http://doc.gitlab.com/ee/integration/jenkins.html). +GitLab Enterprise Edition contains [advanced JIRA support](http://doc.gitlab.com/ee/integration/jira.html) and [advanced Jenkins support](http://doc.gitlab.com/ee/integration/jenkins.html). ## Project services -Integration with services such as Campfire, Flowdock, Gemnasium, HipChat, PivotalTracker and Slack are available in the from of a Project Service. +Integration with services such as Campfire, Flowdock, Gemnasium, HipChat, Pivotal Tracker, and Slack are available in the form of a Project Service. You can find these within GitLab in the Services page under Project Settings if you are at least a master on the project. Project Services are a bit like plugins in that they allow a lot of freedom in adding functionality to GitLab, for example there is also a service that can send an email every time someone pushes new commits. Because GitLab is open source we can ship with the code and tests for all plugins. diff --git a/doc/integration/bitbucket.md b/doc/integration/bitbucket.md new file mode 100644 index 00000000000..cc6389f5aaf --- /dev/null +++ b/doc/integration/bitbucket.md @@ -0,0 +1,121 @@ +# Integrate your server with Bitbucket + +Import projects from Bitbucket and login to your GitLab instance with your Bitbucket account. + +To enable the Bitbucket OmniAuth provider you must register your application with Bitbucket. +Bitbucket will generate an application ID and secret key for you to use. + +1. Sign in to Bitbucket. + +1. Navigate to your individual user settings or a team's settings, depending on how you want the application registered. It does not matter if the application is registered as an individual or a team - that is entirely up to you. + +1. Select "OAuth" in the left menu. + +1. Select "Add consumer". + +1. Provide the required details. + - Name: This can be anything. Consider something like "\<Organization\>'s GitLab" or "\<Your Name\>'s GitLab" or something else descriptive. + - Application description: Fill this in if you wish. + - URL: The URL to your GitLab installation. 'https://gitlab.company.com' +1. Select "Save". + +1. You should now see a Key and Secret in the list of OAuth customers. + Keep this page open as you continue configuration. + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For instalations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "bitbucket", + "app_id" => "YOUR_KEY", + "app_secret" => "YOUR_APP_SECRET", + "url" => "https://bitbucket.org/" + } + ] + ``` + + For installation from source: + + ``` + - { name: 'bitbucket', app_id: 'YOUR_KEY', + app_secret: 'YOUR_APP_SECRET' } + ``` + +1. Change 'YOUR_APP_ID' to the key from the Bitbucket application page from step 7. + +1. Change 'YOUR_APP_SECRET' to the secret from the Bitbucket application page from step 7. + +1. Save the configuration file. + +1. Restart GitLab for the changes to take effect. + +On the sign in page there should now be a Bitbucket icon below the regular sign in form. +Click the icon to begin the authentication process. Bitbucket will ask the user to sign in and authorize the GitLab application. +If everything goes well the user will be returned to GitLab and will be signed in. + +## Bitbucket project import + +To allow projects to be imported directly into GitLab, Bitbucket requires two extra setup steps compared to GitHub and GitLab.com. + +Bitbucket doesn't allow OAuth applications to clone repositories over HTTPS, and instead requires GitLab to use SSH and identify itself using your GitLab server's SSH key. + +### Step 1: Known hosts + +To allow GitLab to connect to Bitbucket over SSH, you need to add 'bitbucket.org' to your GitLab server's known SSH hosts. Take the following steps to do so: + +1. Manually connect to 'bitbucket.org' over SSH, while logged in as the `git` account that GitLab will use: + + ```sh + ssh git@bitbucket.org + ``` + +1. Verify the RSA key fingerprint you'll see in the response matches the one in the [Bitbucket documentation](https://confluence.atlassian.com/display/BITBUCKET/Use+the+SSH+protocol+with+Bitbucket#UsetheSSHprotocolwithBitbucket-KnownhostorBitbucket'spublickeyfingerprints) (the specific IP address doesn't matter): + + ```sh + The authenticity of host 'bitbucket.org (207.223.240.182)' can't be established. + RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40. + Are you sure you want to continue connecting (yes/no)? + ``` + +1. If the fingerprint matches, type `yes` to continue connecting and have 'bitbucket.org' be added to your known hosts. + +1. Your GitLab server is now able to connect to Bitbucket over SSH. Continue to step 2: + +### Step 2: Public key + +To be able to access repositories on Bitbucket, GitLab will automatically register your public key with Bitbucket as a deploy key for the repositories to be imported. Your public key needs to be at `~/.ssh/id_rsa.pub`, which will expand to `/home/git/.ssh/id_rsa.pub` in most configurations. + +If you have that file in place, you're all set and should see the "Import projects from Bitbucket" option enabled. If you don't, do the following: + +1. Create a new SSH key: + + ```sh + sudo -u git -H ssh-keygen + ``` + + Make sure to use an **empty passphrase**. + +2. Restart GitLab to allow it to find the new public key. + +You should now see the "Import projects from Bitbucket" option on the New Project page enabled. diff --git a/doc/integration/external-issue-tracker.md b/doc/integration/external-issue-tracker.md index 87af94512ed..96755707dee 100644 --- a/doc/integration/external-issue-tracker.md +++ b/doc/integration/external-issue-tracker.md @@ -1,13 +1,39 @@ # External issue tracker -GitLab has a great issue tracker but you can also use an external issue tracker such as JIRA, Bugzilla or Redmine. This is something that you can turn on per GitLab project. If for example you configure JIRA it provides the following functionality: +GitLab has a great issue tracker but you can also use an external issue tracker such as Jira, Bugzilla or Redmine. You can configure issue trackers per GitLab project. For instance, if you configure Jira it allows you to do the following: -- the 'Issues' link on the GitLab project pages takes you to the appropriate JIRA issue index; -- clicking 'New issue' on the project dashboard creates a new JIRA issue; -- To reference JIRA issue PROJECT-1234 in comments, use syntax PROJECT-1234. Commit messages get turned into HTML links to the corresponding JIRA issue. +- the 'Issues' link on the GitLab project pages takes you to the appropriate Jira issue index; +- clicking 'New issue' on the project dashboard creates a new Jira issue; +- To reference Jira issue PROJECT-1234 in comments, use syntax PROJECT-1234. Commit messages get turned into HTML links to the corresponding Jira issue. - + -You can configure the integration in the gitlab.yml configuration file. +GitLab Enterprise Edition contains [advanced JIRA support](http://doc.gitlab.com/ee/integration/jira.html). + +## Configuration + +### Project Service + +You can enable an external issue tracker per project. As an example, we will configure `Redmine` for project named gitlab-ci. + +Fill in the required details on the page: + + + +* `description` A name for the issue tracker (to differentiate between instances, for example). +* `project_url` The URL to the project in Redmine which is being linked to this GitLab project. +* `issues_url` The URL to the issue in Redmine project that is linked to this GitLab project. Note that the `issues_url` requires `:id` in the url. This id is used by GitLab as a placeholder to replace the issue number. +* `new_issue_url` This is the URL to create a new issue in Redmine for the project linked to this GitLab project. + +### Service Template + +It is necessary to configure the external issue tracker per project, because project specific details are needed for the integration with GitLab. +The admin can add a service template that sets a default for each project. This makes it much easier to configure individual projects. + +In GitLab Admin section, navigate to `Service Templates` and choose the service template you want to create: + + + +After the template is created, the template details will be pre-filled on the project service page. Support to add your commits to the Jira ticket automatically is [available in GitLab EE](http://doc.gitlab.com/ee/integration/jira.html). diff --git a/doc/integration/github.md b/doc/integration/github.md index 714593d8266..a9f1bc31bb4 100644 --- a/doc/integration/github.md +++ b/doc/integration/github.md @@ -1,6 +1,9 @@ -# GitHub OAuth2 OmniAuth Provider +# Integrate your server with GitHub -To enable the GitHub OmniAuth provider you must register your application with GitHub. GitHub will generate a client ID and secret key for you to use. +Import projects from GitHub and login to your GitLab instance with your GitHub account. + +To enable the GitHub OmniAuth provider you must register your application with GitHub. +GitHub will generate an application ID and secret key for you to use. 1. Sign in to GitHub. @@ -14,35 +17,63 @@ To enable the GitHub OmniAuth provider you must register your application with G - Application name: This can be anything. Consider something like "\<Organization\>'s GitLab" or "\<Your Name\>'s GitLab" or something else descriptive. - Homepage URL: The URL to your GitLab installation. 'https://gitlab.company.com' - Application description: Fill this in if you wish. - - Authorization callback URL: 'https://gitlab.company.com/users/auth/github/callback' + - Authorization callback URL: 'https://gitlab.company.com/' 1. Select "Register application". -1. You should now see a Client ID and Client Secret near the top right of the page (see screenshot). Keep this page open as you continue configuration.  +1. You should now see a Client ID and Client Secret near the top right of the page (see screenshot). + Keep this page open as you continue configuration. +  1. On your GitLab server, open the configuration file. + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For instalations from source: + ```sh - cd /home/git/gitlab + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml + sudo -u git -H editor config/gitlab.yml ``` -1. Find the section dealing with OmniAuth. See [Initial OmniAuth Configuration](README.md#initial-omniauth-configuration) for more details. +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "github", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "url" => "https://github.com/", + "args" => { "scope" => "user:email" } } + } + ] + ``` -1. Under `providers:` uncomment (or add) lines that look like the following: + For installation from source: ``` - - { name: 'github', app_id: 'YOUR APP ID', - app_secret: 'YOUR APP SECRET', - args: { scope: 'user:email' } } + - { name: 'github', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { scope: 'user:email' } } ``` -1. Change 'YOUR APP ID' to the client ID from the GitHub application page from step 7. +1. Change 'YOUR_APP_ID' to the client ID from the GitHub application page from step 7. -1. Change 'YOUR APP SECRET' to the client secret from the GitHub application page from step 7. +1. Change 'YOUR_APP_SECRET' to the client secret from the GitHub application page from step 7. 1. Save the configuration file. 1. Restart GitLab for the changes to take effect. -On the sign in page there should now be a GitHub icon below the regular sign in form. Click the icon to begin the authentication process. GitHub will ask the user to sign in and authorize the GitLab application. If everything goes well the user will be returned to GitLab and will be signed in. +On the sign in page there should now be a GitHub icon below the regular sign in form. +Click the icon to begin the authentication process. GitHub will ask the user to sign in and authorize the GitLab application. +If everything goes well the user will be returned to GitLab and will be signed in.
\ No newline at end of file diff --git a/doc/integration/github_app.png b/doc/integration/github_app.png Binary files differindex c0873b2e20d..d890345ced9 100644 --- a/doc/integration/github_app.png +++ b/doc/integration/github_app.png diff --git a/doc/integration/gitlab.md b/doc/integration/gitlab.md new file mode 100644 index 00000000000..49ffaa62af8 --- /dev/null +++ b/doc/integration/gitlab.md @@ -0,0 +1,84 @@ +# Integrate your server with GitLab.com + +Import projects from GitLab.com and login to your GitLab instance with your GitLab.com account. + +To enable the GitLab.com OmniAuth provider you must register your application with GitLab.com. +GitLab.com will generate an application ID and secret key for you to use. + +1. Sign in to GitLab.com + +1. Navigate to your profile settings. + +1. Select "Applications" in the left menu. + +1. Select "New application". + +1. Provide the required details. + - Name: This can be anything. Consider something like "\<Organization\>'s GitLab" or "\<Your Name\>'s GitLab" or something else descriptive. + - Redirect URI: + + ``` + http://your-gitlab.example.com/import/gitlab/callback + http://your-gitlab.example.com/users/auth/gitlab/callback + ``` + + The first link is required for the importer and second for the authorization. + +1. Select "Submit". + +1. You should now see a Client ID and Client Secret near the top right of the page (see screenshot). + Keep this page open as you continue configuration. +  + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For instalations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "gitlab", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "args" => { "scope" => "api" } } + } + ] + ``` + + For installations from source: + + ``` + - { name: 'gitlab', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { scope: 'api' } } + ``` + +1. Change 'YOUR_APP_ID' to the Application ID from the GitLab.com application page. + +1. Change 'YOUR_APP_SECRET' to the secret from the GitLab.com application page. + +1. Save the configuration file. + +1. Restart GitLab for the changes to take effect. + +On the sign in page there should now be a GitLab.com icon below the regular sign in form. +Click the icon to begin the authentication process. GitLab.com will ask the user to sign in and authorize the GitLab application. +If everything goes well the user will be returned to your GitLab instance and will be signed in.
\ No newline at end of file diff --git a/doc/integration/gitlab_actions.png b/doc/integration/gitlab_actions.png Binary files differnew file mode 100644 index 00000000000..b08f54d137b --- /dev/null +++ b/doc/integration/gitlab_actions.png diff --git a/doc/integration/gitlab_app.png b/doc/integration/gitlab_app.png Binary files differnew file mode 100644 index 00000000000..3f9391a821b --- /dev/null +++ b/doc/integration/gitlab_app.png diff --git a/doc/integration/gitlab_buttons_in_gmail.md b/doc/integration/gitlab_buttons_in_gmail.md new file mode 100644 index 00000000000..a9885cef109 --- /dev/null +++ b/doc/integration/gitlab_buttons_in_gmail.md @@ -0,0 +1,28 @@ +# GitLab buttons in Gmail + +GitLab supports [Google actions in email](https://developers.google.com/gmail/markup/actions/actions-overview). + +If correctly setup, emails that require an action will be marked in Gmail. + + + +To get this functioning, you need to be registered with Google. +[See how to register with google in this document.](https://developers.google.com/gmail/markup/registering-with-google) + +To aid the registering with google, GitLab offers a rake task that will send an email to google whitelisting email address from your GitLab server. + +To check what would be sent to the google email address, run the rake task: + +```bash +bundle exec rake gitlab:mail_google_schema_whitelisting RAILS_ENV=production +``` + +**This will not send the email but give you the output of how the mail will look.** + +Copy the output of the rake task to [google email markup tester](https://www.google.com/webmasters/markup-tester/u/0/) and press "Validate". + +If you receive "No errors detected" message from the tester you can send the email using: + +```bash +bundle exec rake gitlab:mail_google_schema_whitelisting RAILS_ENV=production SEND=true +``` diff --git a/doc/integration/google.md b/doc/integration/google.md index 7a78aff8ea4..d7b741ece69 100644 --- a/doc/integration/google.md +++ b/doc/integration/google.md @@ -27,27 +27,50 @@ To enable the Google OAuth2 OmniAuth provider you must register your application - Authorized redirect URI: 'https://gitlab.example.com/users/auth/google_oauth2/callback' 1. Under the heading "Client ID for web application" you should see a Client ID and Client secret (see screenshot). Keep this page open as you continue configuration.  -1. On your GitLab server, open the configuration file. +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For instalations from source: ```sh - cd /home/git/gitlab + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml + sudo -u git -H editor config/gitlab.yml ``` -1. Find the section dealing with OmniAuth. See [Initial OmniAuth Configuration](README.md#initial-omniauth-configuration) for more details. +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "google_oauth2", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "args" => { "access_type" => "offline", "approval_prompt" => '' } } + } + ] + ``` -1. Under `providers:` uncomment (or add) lines that look like the following: + For installations from source: ``` - - { name: 'google_oauth2', app_id: 'YOUR APP ID', - app_secret: 'YOUR APP SECRET', - args: { access_type: 'offline', approval_prompt: '' } } + - { name: 'google_oauth2', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { access_type: 'offline', approval_prompt: '' } } ``` -1. Change 'YOUR APP ID' to the client ID from the GitHub application page from step 7. +1. Change 'YOUR_APP_ID' to the client ID from the Google Developer page from step 10. -1. Change 'YOUR APP SECRET' to the client secret from the GitHub application page from step 7. +1. Change 'YOUR_APP_SECRET' to the client secret from the Google Developer page from step 10. 1. Save the configuration file. diff --git a/doc/integration/ldap.md b/doc/integration/ldap.md index 56b0d826adb..125ce31b521 100644 --- a/doc/integration/ldap.md +++ b/doc/integration/ldap.md @@ -29,9 +29,9 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server label: 'LDAP' host: '_your_ldap_server' - port: 636 + port: 389 uid: 'sAMAccountName' - method: 'ssl' # "tls" or "ssl" or "plain" + method: 'plain' # "tls" or "ssl" or "plain" bind_dn: '_the_full_dn_of_the_user_you_will_bind_with' password: '_the_password_of_the_bind_user' @@ -76,6 +76,9 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server EOS ``` +If you are getting 'Connection Refused' errors when trying to connect to the LDAP server please double-check the LDAP `port` and `method` settings used by GitLab. +Common combinations are `method: 'plain'` and `port: 389`, OR `method: 'ssl'` and `port: 636`. + If you are using a GitLab installation from source you can find the LDAP settings in `/home/git/gitlab/config/gitlab.yml`: ``` diff --git a/doc/integration/oauth_provider.md b/doc/integration/oauth_provider.md new file mode 100644 index 00000000000..192c321f712 --- /dev/null +++ b/doc/integration/oauth_provider.md @@ -0,0 +1,35 @@ +## GitLab as OAuth2 authentication service provider + +This document is about using GitLab as an OAuth authentication service provider to sign into other services. +If you want to use other OAuth authentication service providers to sign into GitLab please see the [OAuth2 client documentation](../api/oauth2.md) + +OAuth2 provides client applications a 'secure delegated access' to server resources on behalf of a resource owner. Or you can allow users to sign in to your application with their GitLab.com account. +In fact OAuth allows to issue access token to third-party clients by an authorization server, +with the approval of the resource owner, or end-user. +Mostly, OAuth2 is using for SSO (Single sign-on). But you can find a lot of different usages for this functionality. +For example, our feature 'GitLab Importer' is using OAuth protocol to give an access to repositories without sharing user credentials to GitLab.com account. +Also GitLab.com application can be used for authentication to your GitLab instance if needed [GitLab OmniAuth](gitlab.md). + +GitLab has two ways to add new OAuth2 application to an instance, you can add application as regular user and through admin area. So GitLab actually can have an instance-wide and a user-wide applications. There is no defferences between them except the different permission levels. + +### Adding application through profile +Go to your profile section 'Application' and press button 'New Application' + + + +After this you will see application form, where "Name" is arbitrary name, "Redirect URI" is URL in your app where users will be sent after authorization on GitLab.com. + + + +### Authorized application +Every application you authorized will be shown in your "Authorized application" sections. + + + +At any time you can revoke access just clicking button "Revoke" + +### OAuth applications in admin area + +If you want to create application that does not belong to certain user you can create it from admin area + +
\ No newline at end of file diff --git a/doc/integration/oauth_provider/admin_application.png b/doc/integration/oauth_provider/admin_application.png Binary files differnew file mode 100644 index 00000000000..a5f34512aa8 --- /dev/null +++ b/doc/integration/oauth_provider/admin_application.png diff --git a/doc/integration/oauth_provider/application_form.png b/doc/integration/oauth_provider/application_form.png Binary files differnew file mode 100644 index 00000000000..ae135db2627 --- /dev/null +++ b/doc/integration/oauth_provider/application_form.png diff --git a/doc/integration/oauth_provider/authorized_application.png b/doc/integration/oauth_provider/authorized_application.png Binary files differnew file mode 100644 index 00000000000..d3ce05be9cc --- /dev/null +++ b/doc/integration/oauth_provider/authorized_application.png diff --git a/doc/integration/oauth_provider/user_wide_applications.png b/doc/integration/oauth_provider/user_wide_applications.png Binary files differnew file mode 100644 index 00000000000..719e1974068 --- /dev/null +++ b/doc/integration/oauth_provider/user_wide_applications.png diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index 00adae58dfa..24f7b4bb4b4 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -1,18 +1,47 @@ # OmniAuth -GitLab leverages OmniAuth to allow users to sign in using Twitter, GitHub, and other popular services. Configuring +GitLab leverages OmniAuth to allow users to sign in using Twitter, GitHub, and other popular services. -OmniAuth does not prevent standard GitLab authentication or LDAP (if configured) from continuing to work. Users can choose to sign in using any of the configured mechanisms. +Configuring OmniAuth does not prevent standard GitLab authentication or LDAP (if configured) from continuing to work. Users can choose to sign in using any of the configured mechanisms. - [Initial OmniAuth Configuration](#initial-omniauth-configuration) - [Supported Providers](#supported-providers) - [Enable OmniAuth for an Existing User](#enable-omniauth-for-an-existing-user) +- [OmniAuth configuration sample when using Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master#omniauth-google-twitter-github-login) ## Initial OmniAuth Configuration -Before configuring individual OmniAuth providers there are a few global settings that need to be verified. +Before configuring individual OmniAuth providers there are a few global settings that are in common for all providers that we need to consider. -1. Open the configuration file. +- Omniauth needs to be enabled, see details below for example. +- `allow_single_sign_on` defaults to `false`. If `false` users must be created manually or they will not be able to +sign in via OmniAuth. +- `block_auto_created_users` defaults to `true`. If `true` auto created users will be blocked by default and will +have to be unblocked by an administrator before they are able to sign in. +- **Note:** If you set `allow_single_sign_on` to `true` and `block_auto_created_users` to `false` please be aware +that any user on the Internet will be able to successfully sign in to your GitLab without administrative approval. + +If you want to change these settings: + +* **For omnibus package** + + Open the configuration file: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + and change + + ``` + gitlab_rails['omniauth_enabled'] = true + gitlab_rails['omniauth_allow_single_sign_on'] = false + gitlab_rails['block_auto_created_users'] = true + ``` + +* **For installations from source** + + Open the configuration file: ```sh cd /home/git/gitlab @@ -20,13 +49,13 @@ Before configuring individual OmniAuth providers there are a few global settings sudo -u git -H editor config/gitlab.yml ``` -1. Find the section dealing with OmniAuth. The section will look similar to the following. + and change the following section ``` - ## OmniAuth settings + ## OmniAuth settings omniauth: # Allow login via Twitter, Google, etc. using OmniAuth providers - enabled: false + enabled: true # CAUTION! # This allows users to login without having a user account first (default: false). @@ -34,47 +63,15 @@ Before configuring individual OmniAuth providers there are a few global settings allow_single_sign_on: false # Locks down those users until they have been cleared by the admin (default: true). block_auto_created_users: true - - ## Auth providers - # Uncomment the following lines and fill in the data of the auth provider you want to use - # If your favorite auth provider is not listed you can use others: - # see https://github.com/gitlabhq/gitlab-public-wiki/wiki/Custom-omniauth-provider-configurations - # The 'app_id' and 'app_secret' parameters are always passed as the first two - # arguments, followed by optional 'args' which can be either a hash or an array. - providers: - # - { name: 'google_oauth2', app_id: 'YOUR APP ID', - # app_secret: 'YOUR APP SECRET', - # args: { access_type: 'offline', approval_prompt: '' } } - # - { name: 'twitter', app_id: 'YOUR APP ID', - # app_secret: 'YOUR APP SECRET'} - # - { name: 'github', app_id: 'YOUR APP ID', - # app_secret: 'YOUR APP SECRET', - # args: { scope: 'user:email' } } - # - {"name": 'shibboleth', - # args: { shib_session_id_field: "HTTP_SHIB_SESSION_ID", - # shib_application_id_field: "HTTP_SHIB_APPLICATION_ID", - # uid_field: "HTTP_EPPN", - # name_field: "HTTP_CN", - # info_fields: {"email": "HTTP_MAIL" } } } - ``` -1. Change `enabled` to `true`. - -1. Consider the next two configuration options: `allow_single_sign_on` and `block_auto_created_users`. - - - `allow_single_sign_on` defaults to `false`. If `false` users must be created manually or they will not be able to - sign in via OmniAuth. - - `block_auto_created_users` defaults to `true`. If `true` auto created users will be blocked by default and will - have to be unblocked by an administrator before they are able to sign in. - - **Note:** If you set `allow_single_sign_on` to `true` and `block_auto_created_users` to `false` please be aware - that any user on the Internet will be able to successfully sign in to your GitLab without administrative approval. - -1. Choose one or more of the Supported Providers below to continue configuration. +Now we can choose one or more of the Supported Providers below to continue configuration. ## Supported Providers - [GitHub](github.md) +- [Bitbucket](bitbucket.md) +- [GitLab.com](gitlab.md) - [Google](google.md) - [Shibboleth](shibboleth.md) - [Twitter](twitter.md) diff --git a/doc/integration/redmine_configuration.png b/doc/integration/redmine_configuration.png Binary files differnew file mode 100644 index 00000000000..6b145363229 --- /dev/null +++ b/doc/integration/redmine_configuration.png diff --git a/doc/integration/redmine_service_template.png b/doc/integration/redmine_service_template.png Binary files differnew file mode 100644 index 00000000000..1159eb5b964 --- /dev/null +++ b/doc/integration/redmine_service_template.png diff --git a/doc/integration/shibboleth.md b/doc/integration/shibboleth.md index 78317a5c0f2..6258e5f1030 100644 --- a/doc/integration/shibboleth.md +++ b/doc/integration/shibboleth.md @@ -2,19 +2,19 @@ This documentation is for enabling shibboleth with gitlab-omnibus package. -In order to enable Shibboleth support in gitlab we need to use Apache instead of Nginx (It may be possible to use Nginx, however I did not found way to easily configure nginx that is bundled in gitlab-omnibus package). Apache uses mod_shib2 module for shibboleth authentication and can pass attributes as headers to omniauth-shibboleth provider. +In order to enable Shibboleth support in gitlab we need to use Apache instead of Nginx (It may be possible to use Nginx, however I did not found way to easily configure Nginx that is bundled in gitlab-omnibus package). Apache uses mod_shib2 module for shibboleth authentication and can pass attributes as headers to omniauth-shibboleth provider. To enable the Shibboleth OmniAuth provider you must: -1. Configure Apache shibboleth module. Installation and configuration of module it self is out of scope of this document. +1. Configure Apache shibboleth module. Installation and configuration of module it self is out of scope of this document. Check https://wiki.shibboleth.net/ for more info. -1. You can find Apache config in gitlab-reciepes (https://github.com/gitlabhq/gitlab-recipes/blob/master/web-server/apache/gitlab-ssl.conf) +1. You can find Apache config in gitlab-recipes (https://github.com/gitlabhq/gitlab-recipes/blob/master/web-server/apache/gitlab-ssl.conf) Following changes are needed to enable shibboleth: -protect omniauth-shibboleth callback url: +protect omniauth-shibboleth callback URL: ``` <Location /users/auth/shibboleth/callback> AuthType shibboleth @@ -32,25 +32,25 @@ protect omniauth-shibboleth callback url: SetHandler shib </Location> ``` -exclude shibboleth urls from rewriting, add "RewriteCond %{REQUEST_URI} !/Shibboleth.sso" and "RewriteCond %{REQUEST_URI} !/shibboleth-sp", config should look like this: +exclude shibboleth URLs from rewriting, add "RewriteCond %{REQUEST_URI} !/Shibboleth.sso" and "RewriteCond %{REQUEST_URI} !/shibboleth-sp", config should look like this: ``` - #apache equivalent of nginx try files + # Apache equivalent of Nginx try files RewriteEngine on RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f - RewriteCond %{REQUEST_URI} !/Shibboleth.sso - RewriteCond %{REQUEST_URI} !/shibboleth-sp + RewriteCond %{REQUEST_URI} !/Shibboleth.sso + RewriteCond %{REQUEST_URI} !/shibboleth-sp RewriteRule .* http://127.0.0.1:8080%{REQUEST_URI} [P,QSA] RequestHeader set X_FORWARDED_PROTO 'https' ``` -1. Edit /etc/gitlab/gitlab.rb configuration file, your shibboleth attributes should be in form of "HTTP_ATTRIBUTE" and you should addjust them to your need and environment. Add any other configuration you need. +1. Edit /etc/gitlab/gitlab.rb configuration file, your shibboleth attributes should be in form of "HTTP_ATTRIBUTE" and you should addjust them to your need and environment. Add any other configuration you need. -File it should look like this: +File should look like this: ``` external_url 'https://gitlab.example.com' gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' -# disable nginx +# disable Nginx nginx['enable'] = false gitlab_rails['omniauth_allow_single_sign_on'] = true @@ -70,7 +70,7 @@ gitlab_rails['omniauth_providers'] = [ ] ``` -1. Save changes and reconfigure gitlab: +1. Save changes and reconfigure gitlab: ``` sudo gitlab-ctl reconfigure ``` diff --git a/doc/integration/slack.md b/doc/integration/slack.md index f2e73f272ef..2fd22c513ad 100644 --- a/doc/integration/slack.md +++ b/doc/integration/slack.md @@ -35,7 +35,7 @@ After Slack is ready we need to setup GitLab. Here are the steps to achieve this 1. Fill in your Slack details - Mark it as active - - Paste in the webhook url you got from Slack + - Paste in the webhook URL you got from Slack Have fun :) diff --git a/doc/integration/twitter.md b/doc/integration/twitter.md index d1b52927d30..fe9091ad9a8 100644 --- a/doc/integration/twitter.md +++ b/doc/integration/twitter.md @@ -13,7 +13,7 @@ To enable the Twitter OmniAuth provider you must register your application with something else descriptive. - Description: Create a description. - Website: The URL to your GitLab installation. 'https://gitlab.example.com' - - Callback URL: 'https://gitlab.example.com/users/auth/github/callback' + - Callback URL: 'https://gitlab.example.com/users/auth/twitter/callback' - Agree to the "Rules of the Road."  @@ -33,25 +33,46 @@ To enable the Twitter OmniAuth provider you must register your application with 1. On your GitLab server, open the configuration file. + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For instalations from source: + ```sh - cd /home/git/gitlab + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml + sudo -u git -H editor config/gitlab.yml ``` -1. Find the section dealing with OmniAuth. See [Initial OmniAuth Configuration](README.md#initial-omniauth-configuration) -for more details. +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "twitter", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET" + } + ] + ``` -1. Under `providers:` uncomment (or add) lines that look like the following: + For installations from source: ``` - - { name: 'twitter', app_id: 'YOUR APP ID', - app_secret: 'YOUR APP SECRET' } + - { name: 'twitter', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET' } ``` -1. Change 'YOUR APP ID' to the API key from Twitter page in step 11. +1. Change 'YOUR_APP_ID' to the API key from Twitter page in step 11. -1. Change 'YOUR APP SECRET' to the API secret from the Twitter page in step 11. +1. Change 'YOUR_APP_SECRET' to the API secret from the Twitter page in step 11. 1. Save the configuration file. diff --git a/doc/logs/logs.md b/doc/logs/logs.md new file mode 100644 index 00000000000..07302894dd4 --- /dev/null +++ b/doc/logs/logs.md @@ -0,0 +1,100 @@ +## Log system +GitLab has advanced log system so everything is logging and you can analize your instance using various system log files. +These log files are typically plain text in a standard log file format. This guide talks about how to read and use these system log files. + +#### production.log +This file lives in `/var/log/gitlab/gitlab-rails/production.log` for omnibus package or in `/home/git/gitlab/logs/production.log` for installations from the source. + +This file contains information about all performed requests. You can see url and type of request, IP address and what exactly parts of code were involved to service this particular request. Also you can see all SQL request that have been performed and how much time it took. +This task is more useful for GitLab contributors and developers. Use part of this log file when you are going to report bug. + +``` +Started GET "/gitlabhq/yaml_db/tree/master" for 168.111.56.1 at 2015-02-12 19:34:53 +0200 +Processing by Projects::TreeController#show as HTML + Parameters: {"project_id"=>"gitlabhq/yaml_db", "id"=>"master"} + + ... [CUT OUT] + + amespaces"."created_at" DESC, "namespaces"."id" DESC LIMIT 1[0m [["id", 26]] + [1m[35mCACHE (0.0ms)[0m SELECT "members".* FROM "members" WHERE "members"."source_type" = 'Project' AND "members"."type" IN ('ProjectMember') AND "members"."source_id" = $1 AND "members"."source_type" = $2 AND "members"."user_id" = 1 ORDER BY "members"."created_at" DESC, "members"."id" DESC LIMIT 1 [["source_id", 18], ["source_type", "Project"]] + [1m[36mCACHE (0.0ms)[0m [1mSELECT "members".* FROM "members" WHERE "members"."source_type" = 'Project' AND "members". + [1m[36m (1.4ms)[0m [1mSELECT COUNT(*) FROM "merge_requests" WHERE "merge_requests"."target_project_id" = $1 AND ("merge_requests"."state" IN ('opened','reopened'))[0m [["target_project_id", 18]] + Rendered layouts/nav/_project.html.haml (28.0ms) + Rendered layouts/_collapse_button.html.haml (0.2ms) + Rendered layouts/_flash.html.haml (0.1ms) + Rendered layouts/_page.html.haml (32.9ms) +Completed 200 OK in 166ms (Views: 117.4ms | ActiveRecord: 27.2ms) +``` +In this example we can see that server processed HTTP request with url `/gitlabhq/yaml_db/tree/master` from IP 168.111.56.1 at 2015-02-12 19:34:53 +0200. Also we can see that request was processed by Projects::TreeController. + +#### application.log +This file lives in `/var/log/gitlab/gitlab-rails/application.log` for omnibus package or in `/home/git/gitlab/logs/application.log` for installations from the source. + +This log file helps you discover events happening in your instance such as user creation, project removing and so on. + +``` +October 06, 2014 11:56: User "Administrator" (admin@example.com) was created +October 06, 2014 11:56: Documentcloud created a new project "Documentcloud / Underscore" +October 06, 2014 11:56: Gitlab Org created a new project "Gitlab Org / Gitlab Ce" +October 07, 2014 11:25: User "Claudie Hodkiewicz" (nasir_stehr@olson.co.uk) was removed +October 07, 2014 11:25: Project "project133" was removed +``` +#### githost.log +This file lives in `/var/log/gitlab/gitlab-rails/githost.log` for omnibus package or in `/home/git/gitlab/logs/githost.log` for installations from the source. + +The GitLab has to interact with git repositories but in some rare cases something can go wrong and in this case you will know what exactly happened. This log file contains all failed requests from GitLab to git repository. In majority of cases this file will be useful for developers only. +``` +December 03, 2014 13:20 -> ERROR -> Command failed [1]: /usr/bin/git --git-dir=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq/.git --work-tree=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq merge --no-ff -mMerge branch 'feature_conflict' into 'feature' source/feature_conflict + +error: failed to push some refs to '/Users/vsizov/gitlab-development-kit/repositories/gitlabhq/gitlab_git.git' +``` + +#### satellites.log +This file lives in `/var/log/gitlab/gitlab-rails/satellites.log` for omnibus package or in `/home/git/gitlab/logs/satellites.log` for installations from the source. + +In some cases GitLab should perform write actions to git repository, for example when it is needed to merge the merge request or edit a file with online editor. If something went wrong you can look into this file to find out what exactly happened. +``` +October 07, 2014 11:36: Failed to create satellite for Chesley Weimann III / project1817 +October 07, 2014 11:36: PID: 1872: git clone /Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/repositories/conrad6841/gitlabhq.git /Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/conrad6841/gitlabhq +October 07, 2014 11:36: PID: 1872: -> fatal: repository '/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/repositories/conrad6841/gitlabhq.git' does not exist +``` + +#### sidekiq.log +This file lives in `/var/log/gitlab/gitlab-rails/sidekiq.log` for omnibus package or in `/home/git/gitlab/logs/sidekiq.log` for installations from the source. + +GitLab uses background jobs for processing tasks which can take a long time. All information about processing these jobs are writing down to this file. +``` +2014-06-10T07:55:20Z 2037 TID-tm504 ERROR: /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/1.9.1/gems/redis-3.0.7/lib/redis/client.rb:228:in `read' +2014-06-10T18:18:26Z 14299 TID-55uqo INFO: Booting Sidekiq 3.0.0 with redis options {:url=>"redis://localhost:6379/0", :namespace=>"sidekiq"} +``` + +#### gitlab-shell.log +This file lives in `/var/log/gitlab/gitlab-shell/gitlab-shell.log` for omnibus package or in `/home/git/gitlab-shell/logs/sidekiq.log` for installations from the source. + +gitlab-shell is using by Gitlab for executing git commands and provide ssh access to git repositories. + +``` +I, [2015-02-13T06:17:00.671315 #9291] INFO -- : Adding project root/example.git at </var/opt/gitlab/git-data/repositories/root/dcdcdcdcd.git>. +I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory and simlinking global hooks directory for /var/opt/gitlab/git-data/repositories/root/example.git. +``` + +#### unicorn_stderr.log +This file lives in `/var/log/gitlab/unicorn/unicorn_stderr.log` for omnibus package or in `/home/git/gitlab/logs/unicorn_stderr.log` for installations from the source. + +Unicorn is a high-performance forking Web server which is used for serving GitLab application. You can look at this log, for example, if your application does not respond. This log cantains all information about state of unicorn processes at any given time. + +``` +I, [2015-02-13T06:14:46.680381 #9047] INFO -- : Refreshing Gem list +I, [2015-02-13T06:14:56.931002 #9047] INFO -- : listening on addr=127.0.0.1:8080 fd=12 +I, [2015-02-13T06:14:56.931381 #9047] INFO -- : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=13 +I, [2015-02-13T06:14:56.936638 #9047] INFO -- : master process ready +I, [2015-02-13T06:14:56.946504 #9092] INFO -- : worker=0 spawned pid=9092 +I, [2015-02-13T06:14:56.946943 #9092] INFO -- : worker=0 ready +I, [2015-02-13T06:14:56.947892 #9094] INFO -- : worker=1 spawned pid=9094 +I, [2015-02-13T06:14:56.948181 #9094] INFO -- : worker=1 ready +W, [2015-02-13T07:16:01.312916 #9094] WARN -- : #<Unicorn::HttpServer:0x0000000208f618>: worker (pid: 9094) exceeds memory limit (320626688 bytes > 247066940 bytes) +W, [2015-02-13T07:16:01.313000 #9094] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 9094) alive: 3621 sec (trial 1) +I, [2015-02-13T07:16:01.530733 #9047] INFO -- : reaped #<Process::Status: pid 9094 exit 0> worker=1 +I, [2015-02-13T07:16:01.534501 #13379] INFO -- : worker=1 spawned pid=13379 +I, [2015-02-13T07:16:01.534848 #13379] INFO -- : worker=1 ready +``` diff --git a/doc/markdown/markdown.md b/doc/markdown/markdown.md index edb7a975503..1096ea9656c 100644 --- a/doc/markdown/markdown.md +++ b/doc/markdown/markdown.md @@ -6,7 +6,7 @@ * [Newlines](#newlines) * [Multiple underscores in words](#multiple-underscores-in-words) -* [URL autolinking](#url-autolinking) +* [URL auto-linking](#url-autolinking) * [Code and Syntax Highlighting](#code-and-syntax-highlighting) * [Emoji](#emoji) * [Special GitLab references](#special-gitlab-references) @@ -40,7 +40,7 @@ You can use GFM in - milestones - wiki pages -You can also use other rich text files in GitLab. You might have to install a depency to do so. Please see the [github-markup gem readme](https://github.com/gitlabhq/markup#markups) for more information. +You can also use other rich text files in GitLab. You might have to install a dependency to do so. Please see the [github-markup gem readme](https://github.com/gitlabhq/markup#markups) for more information. ## Newlines @@ -68,7 +68,7 @@ It is not reasonable to italicize just _part_ of a word, especially when you're perform_complicated_task do_this_and_do_that_and_another_thing -## URL autolinking +## URL auto-linking GFM will autolink standard URLs you copy and paste into your text. So if you want to link to a URL (instead of a textural link), you can simply put the URL in verbatim and it will be turned into a link to that URL. @@ -148,7 +148,7 @@ But let's throw in a <b>tag</b>. If you are new to this, don't be :fearful_face:. You can easily join the emoji :family:. All you need to do is to look up on the supported codes. - Consult the [Emoji Cheat Sheet](https://www.dropbox.com/s/b9xaqb977s6d8w1/cheat_sheet.pdf) for a list of all supported emoji codes. :thumbsup: + Consult the [Emoji Cheat Sheet](https://s3.amazonaws.com/emoji-cheatsheet/cheat_sheet.pdf) for a list of all supported emoji codes. :thumbsup: Sometimes you want to be a :ninja: and add some :glowing_star: to your :speech_balloon:. Well we have a gift for you: @@ -158,7 +158,7 @@ You can use it to point out a :bug: or warn about :speak_no_evil_monkey: patches If you are new to this, don't be :fearful_face:. You can easily join the emoji :family:. All you need to do is to look up on the supported codes. -Consult the [Emoji Cheat Sheet](https://www.dropbox.com/s/b9xaqb977s6d8w1/cheat_sheet.pdf) for a list of all supported emoji codes. :thumbsup: +Consult the [Emoji Cheat Sheet](https://s3.amazonaws.com/emoji-cheatsheet/cheat_sheet.pdf) for a list of all supported emoji codes. :thumbsup: ## Special GitLab References @@ -170,7 +170,7 @@ GFM will turn that reference into a link so you can navigate between them easily GFM will recognize the following: -- @foo : for team members +- @foo : for specific team members or groups - @all : for the whole team - #123 : for issues - !123 : for merge requests @@ -250,17 +250,17 @@ The IDs are generated from the content of the header according to the following For example: ``` -###### ..Ab_c-d. e [anchor](url) .. +###### ..Ab_c-d. e [anchor](URL) .. ``` which renders as: -###### ..Ab_c-d. e [anchor](url) .. +###### ..Ab_c-d. e [anchor](URL) .. will first be converted by step 1) into a string like: ``` -..Ab_c-d. e <a href="url">anchor</a> <img src="url" alt="alt text"/>.. +..Ab_c-d. e <a href="URL">anchor</a> <img src="URL" alt="alt text"/>.. ``` After removing the tags in step 2) we get: @@ -277,8 +277,8 @@ ab_c-d-e-anchor Note in particular how: -- for markdown anchors `[text](url)`, only the `text` is used -- markdown images `` are completely ignored +- for markdown anchors `[text](URL)`, only the `text` is used +- markdown images `` are completely ignored ## Emphasis @@ -420,6 +420,8 @@ Quote break. You can also use raw HTML in your Markdown, and it'll mostly work pretty well. +Note that inline HTML is disabled in the default Gitlab configuration, although it is [possible](https://github.com/gitlabhq/gitlabhq/pull/8007/commits) for the system administrator to enable it. + ```no-highlight <dl> <dt>Definition list</dt> diff --git a/doc/operations/README.md b/doc/operations/README.md new file mode 100644 index 00000000000..f1456c6c8e2 --- /dev/null +++ b/doc/operations/README.md @@ -0,0 +1,4 @@ +# GitLab operations + +- [Sidekiq MemoryKiller](sidekiq_memory_killer.md) +- [Cleaning up Redis sessions](cleaning_up_redis_sessions.md) diff --git a/doc/operations/cleaning_up_redis_sessions.md b/doc/operations/cleaning_up_redis_sessions.md new file mode 100644 index 00000000000..93521e976d5 --- /dev/null +++ b/doc/operations/cleaning_up_redis_sessions.md @@ -0,0 +1,52 @@ +# Cleaning up stale Redis sessions + +Since version 6.2, GitLab stores web user sessions as key-value pairs in Redis. +Prior to GitLab 7.3, user sessions did not automatically expire from Redis. If +you have been running a large GitLab server (thousands of users) since before +GitLab 7.3 we recommend cleaning up stale sessions to compact the Redis +database after you upgrade to GitLab 7.3. You can also perform a cleanup while +still running GitLab 7.2 or older, but in that case new stale sessions will +start building up again after you clean up. + +In GitLab versions prior to 7.3.0, the session keys in Redis are 16-byte +hexadecimal values such as '976aa289e2189b17d7ef525a6702ace9'. Starting with +GitLab 7.3.0, the keys are +prefixed with 'session:gitlab:', so they would look like +'session:gitlab:976aa289e2189b17d7ef525a6702ace9'. Below we describe how to +remove the keys in the old format. + +First we define a shell function with the proper Redis connection details. + +``` +rcli() { + # This example works for Omnibus installations of GitLab 7.3 or newer. For an + # installation from source you will have to change the socket path and the + # path to redis-cli. + sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@" +} + +# test the new shell function; the response should be PONG +rcli ping +``` + +Now we do a search to see if there are any session keys in the old format for +us to clean up. + +``` +# returns the number of old-format session keys in Redis +rcli keys '*' | grep '^[a-f0-9]\{32\}$' | wc -l +``` + +If the number is larger than zero, you can proceed to expire the keys from +Redis. If the number is zero there is nothing to clean up. + +``` +# Tell Redis to expire each matched key after 600 seconds. +rcli keys '*' | grep '^[a-f0-9]\{32\}$' | awk '{ print "expire", $0, 600 }' | rcli +# This will print '(integer) 1' for each key that gets expired. +``` + +Over the next 15 minutes (10 minutes expiry time plus 5 minutes Redis +background save interval) your Redis database will be compacted. If you are +still using GitLab 7.2, users who are not clicking around in GitLab during the +10 minute expiry window will be signed out of GitLab. diff --git a/doc/operations/sidekiq_memory_killer.md b/doc/operations/sidekiq_memory_killer.md new file mode 100644 index 00000000000..867b01b0d5a --- /dev/null +++ b/doc/operations/sidekiq_memory_killer.md @@ -0,0 +1,38 @@ +# Sidekiq MemoryKiller + +The GitLab Rails application code suffers from memory leaks. For web requests +this problem is made manageable using +[unicorn-worker-killer](https://github.com/kzk/unicorn-worker-killer) which +restarts Unicorn worker processes in between requests when needed. The Sidekiq +MemoryKiller applies the same approach to the Sidekiq processes used by GitLab +to process background jobs. + +Unlike unicorn-worker-killer, which is enabled by default for all GitLab +installations since GitLab 6.4, the Sidekiq MemoryKiller is enabled by default +_only_ for Omnibus packages. The reason for this is that the MemoryKiller +relies on Runit to restart Sidekiq after a memory-induced shutdown and GitLab +installations from source do not all use Runit or an equivalent. + +With the default settings, the MemoryKiller will cause a Sidekiq restart no +more often than once every 15 minutes, with the restart causing about one +minute of delay for incoming background jobs. + +## Configuring the MemoryKiller + +The MemoryKiller is controlled using environment variables. + +- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is + greater than 0, then after each Sidekiq job, the MemoryKiller will check the + RSS of the Sidekiq process that executed the job. If the RSS of the Sidekiq + process (expressed in kilobytes) exceeds SIDEKIQ_MEMORY_KILLER_MAX_RSS, a + delayed shutdown is triggered. The default value for Omnibus packages is set + [in the omnibus-gitlab + repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb). +- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults 900 seconds (15 minutes). When + a shutdown is triggered, the Sidekiq process will keep working normally for + another 15 minutes. +- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. When the grace + time has expired, the MemoryKiller tells Sidekiq to stop accepting new jobs. + Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells + Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must + restart Sidekiq. diff --git a/doc/permissions/permissions.md b/doc/permissions/permissions.md index d561868c8bb..c9928e11b2e 100644 --- a/doc/permissions/permissions.md +++ b/doc/permissions/permissions.md @@ -8,7 +8,6 @@ If a user is a GitLab administrator they receive all permissions. ## Project - | Action | Guest | Reporter | Developer | Master | Owner | |---------------------------------------|---------|------------|-------------|----------|--------| | Create new issue | ✓ | ✓ | ✓ | ✓ | ✓ | @@ -19,6 +18,7 @@ If a user is a GitLab administrator they receive all permissions. | Create new merge request | | | ✓ | ✓ | ✓ | | Create new branches | | | ✓ | ✓ | ✓ | | Push to non-protected branches | | | ✓ | ✓ | ✓ | +| Force push to non-protected branches | | | ✓ | ✓ | ✓ | | Remove non-protected branches | | | ✓ | ✓ | ✓ | | Add tags | | | ✓ | ✓ | ✓ | | Write a wiki | | | ✓ | ✓ | ✓ | @@ -28,6 +28,7 @@ If a user is a GitLab administrator they receive all permissions. | Add new team members | | | | ✓ | ✓ | | Push to protected branches | | | | ✓ | ✓ | | Enable/disable branch protection | | | | ✓ | ✓ | +| Turn on/off prot. branch push for devs| | | | ✓ | ✓ | | Rewrite/remove git tags | | | | ✓ | ✓ | | Edit project | | | | ✓ | ✓ | | Add deploy keys to project | | | | ✓ | ✓ | @@ -35,6 +36,8 @@ If a user is a GitLab administrator they receive all permissions. | Switch visibility level | | | | | ✓ | | Transfer project to another namespace | | | | | ✓ | | Remove project | | | | | ✓ | +| Force push to protected branches | | | | | | +| Remove protected branches | | | | | | ## Group @@ -46,4 +49,4 @@ If a user is a GitLab administrator they receive all permissions. | Manage group members | | | | | ✓ | | Remove group | | | | | ✓ | -Any user can remove himself from a group, unless he is the last Owner of the group. +Any user can remove themselves from a group, unless they are the last Owner of the group. diff --git a/doc/project_services/project_services.md b/doc/project_services/project_services.md index 20a69a211dd..93a57485cfd 100644 --- a/doc/project_services/project_services.md +++ b/doc/project_services/project_services.md @@ -4,15 +4,16 @@ __Project integrations with external services for continuous integration and mor ## Services -- Assemblia -- [Atlassian Bamboo CI](bamboo.md) An Atlassian product for continous integration. +- Assembla +- [Atlassian Bamboo CI](bamboo.md) An Atlassian product for continuous integration. - Build box - Campfire - Emails on push - Flowdock - Gemnasium - GitLab CI -- Hipchat -- PivotalTracker +- HipChat +- Pivotal Tracker - Pushover - Slack +- TeamCity diff --git a/doc/raketasks/README.md b/doc/raketasks/README.md index 9e2f697bca6..770b7a70fe0 100644 --- a/doc/raketasks/README.md +++ b/doc/raketasks/README.md @@ -1,5 +1,8 @@ +# Rake tasks + - [Backup restore](backup_restore.md) - [Cleanup](cleanup.md) +- [Features](features.md) - [Maintenance](maintenance.md) and self-checks - [User management](user_management.md) - [Web hooks](web_hooks.md) diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index b4581e2a07a..99cdfff0ac6 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -13,8 +13,8 @@ You can only restore a backup to exactly the same version of GitLab that you cre # use this command if you've installed GitLab with the Omnibus package sudo gitlab-rake gitlab:backup:create -# if you've installed GitLab from source or using the cookbook -bundle exec rake gitlab:backup:create RAILS_ENV=production +# if you've installed GitLab from source +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ``` Example output: @@ -137,7 +137,7 @@ with the name of your bucket: Please be informed that a backup does not store your configuration files. If you use an Omnibus package please see the [instructions in the readme to backup your configuration](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#backup-and-restore-omnibus-gitlab-configuration). If you have a cookbook installation there should be a copy of your configuration in Chef. -If you have a manual installation please consider backing up your gitlab.yml file and any SSL keys and certificates. +If you have an installation from source, please consider backing up your `gitlab.yml` file, any SSL keys and certificates, and your [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). ## Restore a previously created backup @@ -147,7 +147,7 @@ You can only restore a backup to exactly the same version of GitLab that you cre # Omnibus package installation sudo gitlab-rake gitlab:backup:restore -# installation from source or cookbook +# installation from source bundle exec rake gitlab:backup:restore RAILS_ENV=production ``` @@ -192,7 +192,7 @@ Deleting tmp directories...[DONE] For Omnibus package installations, see https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#scheduling-a-backup . -For installation from source or cookbook: +For installation from source: ``` cd /home/git/gitlab sudo -u git -H editor config/gitlab.yml # Enable keep_time in the backup section to automatically delete old backups @@ -203,5 +203,31 @@ Add the following lines at the bottom: ``` # Create a full backup of the GitLab repositories and SQL database every day at 4am -0 4 * * * cd /home/git/gitlab && PATH=/usr/local/bin:/usr/bin:/bin bundle exec rake gitlab:backup:create RAILS_ENV=production +0 4 * * * cd /home/git/gitlab && PATH=/usr/local/bin:/usr/bin:/bin bundle exec rake gitlab:backup:create RAILS_ENV=production CRON=1 ``` + +The `CRON=1` environment setting tells the backup script to suppress all progress output if there are no errors. +This is recommended to reduce cron spam. + +## Alternative backup strategies + +If your GitLab server contains a lot of Git repository data you may find the GitLab backup script to be too slow. +In this case you can consider using filesystem snapshots as part of your backup strategy. + +Example: Amazon EBS + +> A GitLab server using omnibus-gitlab hosted on Amazon AWS. +> An EBS drive containing an ext4 filesystem is mounted at `/var/opt/gitlab`. +> In this case you could make an application backup by taking an EBS snapshot. +> The backup includes all repositories, uploads and Postgres data. + +Example: LVM snapshots + rsync + +> A GitLab server using omnibus-gitlab, with an LVM logical volume mounted at `/var/opt/gitlab`. +> Replicating the `/var/opt/gitlab` directory using rsync would not be reliable because too many files would change while rsync is running. +> Instead of rsync-ing `/var/opt/gitlab`, we create a temporary LVM snapshot, which we mount as a read-only filesystem at `/mnt/gitlab_backup`. +> Now we can have a longer running rsync job which will create a consistent replica on the remote server. +> The replica includes all repositories, uploads and Postgres data. + +If you are running GitLab on a virtualized server you can possibly also create VM snapshots of the entire GitLab server. +It is not uncommon however for a VM snapshot to require you to power down the server, so this approach is probably of limited practical use. diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index 9e48f56c951..96d67f7b5d6 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -8,7 +8,7 @@ Remove namespaces(dirs) from `/home/git/repositories` if they don't exist in Git # omnibus-gitlab sudo gitlab-rake gitlab:cleanup:dirs -# installation from source or cookbook +# installation from source bundle exec rake gitlab:cleanup:dirs RAILS_ENV=production ``` @@ -18,6 +18,6 @@ Remove repositories (global only for now) from `/home/git/repositories` if they # omnibus-gitlab sudo gitlab-rake gitlab:cleanup:repos -# installation from source or cookbook +# installation from source bundle exec rake gitlab:cleanup:repos RAILS_ENV=production ``` diff --git a/doc/raketasks/features.md b/doc/raketasks/features.md index 99b3d5525b0..f9a46193547 100644 --- a/doc/raketasks/features.md +++ b/doc/raketasks/features.md @@ -6,7 +6,7 @@ This command will enable the namespaces feature introduced in v4.0. It will move Note: -- Because the **repository location will change**, you will need to **update all your git url's** to point to the new location. +- Because the **repository location will change**, you will need to **update all your git URLs** to point to the new location. - Username can be changed at [Profile / Account](/profile/account) **Example:** diff --git a/doc/raketasks/import.md b/doc/raketasks/import.md index bb229e8acbb..8a38937062e 100644 --- a/doc/raketasks/import.md +++ b/doc/raketasks/import.md @@ -13,15 +13,32 @@ - For omnibus-gitlab, it is located at: `/var/opt/gitlab/git-data/repositories` by default, unless you changed it in the `/etc/gitlab/gitlab.rb` file. -- For manual installations, it is usually located at: `/home/git/repositories` or you can see where +- For installations from source, it is usually located at: `/home/git/repositories` or you can see where your repositories are located by looking at `config/gitlab.yml` under the `gitlab_shell => repos_path` entry. +New folder needs to have git user ownership and read/write/execute access for git user and its group: + +``` +sudo -u git mkdir /var/opt/gitlab/git-data/repositories/new_group +``` + +If you are using an installation from source, replace `/var/opt/gitlab/git-data` +with `/home/git`. + ### Copy your bare repositories inside this newly created folder: ``` -$ cp -r /old/git/foo.git/ /home/git/repositories/new_group/ +sudo cp -r /old/git/foo.git /var/opt/gitlab/git-data/repositories/new_group/ + +# Do this once when you are done copying git repositories +sudo chown -R git:git /var/opt/gitlab/git-data/repositories/new_group/ ``` +`foo.git` needs to be owned by the git user and git users group. + +If you are using an installation from source, replace `/var/opt/gitlab/git-data` +with `/home/git`. + ### Run the command below depending on your type of installation: #### Omnibus Installation @@ -30,7 +47,7 @@ $ cp -r /old/git/foo.git/ /home/git/repositories/new_group/ $ sudo gitlab-rake gitlab:import:repos ``` -#### Manual Installation +#### Installation from source Before running this command you need to change the directory to where your GitLab installation is located: diff --git a/doc/raketasks/maintenance.md b/doc/raketasks/maintenance.md index f6bd7565799..41a994f3f68 100644 --- a/doc/raketasks/maintenance.md +++ b/doc/raketasks/maintenance.md @@ -8,7 +8,7 @@ This command gathers information about your GitLab installation and the System i # omnibus-gitlab sudo gitlab-rake gitlab:env:info -# installation from source or cookbook +# installation from source bundle exec rake gitlab:env:info RAILS_ENV=production ``` @@ -16,30 +16,31 @@ Example output: ``` System information -System: Debian 6.0.7 -Current User: git -Using RVM: no -Ruby Version: 2.0.0-p481 -Gem Version: 1.8.23 -Bundler Version:1.3.5 -Rake Version: 10.0.4 +System: Debian 7.8 +Current User: git +Using RVM: no +Ruby Version: 2.1.5p273 +Gem Version: 2.4.3 +Bundler Version: 1.7.6 +Rake Version: 10.3.2 +Sidekiq Version: 2.17.8 GitLab information -Version: 5.1.0.beta2 -Revision: 4da8b37 -Directory: /home/git/gitlab -DB Adapter: mysql2 -URL: http://example.com -HTTP Clone URL: http://example.com/some-project.git -SSH Clone URL: git@example.com:some-project.git -Using LDAP: no -Using Omniauth: no +Version: 7.7.1 +Revision: 41ab9e1 +Directory: /home/git/gitlab +DB Adapter: postgresql +URL: https://gitlab.example.com +HTTP Clone URL: https://gitlab.example.com/some-project.git +SSH Clone URL: git@gitlab.example.com:some-project.git +Using LDAP: no +Using Omniauth: no GitLab Shell -Version: 1.2.0 -Repositories: /home/git/repositories/ -Hooks: /home/git/gitlab-shell/hooks/ -Git: /usr/bin/git +Version: 2.4.1 +Repositories: /home/git/repositories/ +Hooks: /home/git/gitlab-shell/hooks/ +Git: /usr/bin/git ``` ## Check GitLab configuration @@ -59,7 +60,7 @@ You may also have a look at our [Trouble Shooting Guide](https://github.com/gitl # omnibus-gitlab sudo gitlab-rake gitlab:check -# installation from source or cookbook +# installation from source bundle exec rake gitlab:check RAILS_ENV=production ``` @@ -122,3 +123,56 @@ sudo -u git -H mkdir -p /home/git/gitlab-satellites sudo -u git -H bundle exec rake gitlab:satellites:create RAILS_ENV=production sudo chmod u+rwx,g=rx,o-rwx /home/git/gitlab-satellites ``` + +## Rebuild authorized_keys file + +In some case it is necessary to rebuild the `authorized_keys` file. + +For Omnibus-packages: +``` +sudo gitlab-rake gitlab:shell:setup +``` + +For installations from source: +``` +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:shell:setup RAILS_ENV=production +``` + +``` +This will rebuild an authorized_keys file. +You will lose any data stored in authorized_keys file. +Do you want to continue (yes/no)? yes +``` + +## Clear redis cache + +If for some reason the dashboard shows wrong information you might want to +clear Redis' cache. + +For Omnibus-packages: +``` +sudo gitlab-rake cache:clear +``` + +For installations from source: +``` +cd /home/git/gitlab +sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production +``` + +## Precompile the assets + +Sometimes during version upgrades you might end up with some wrong CSS or +missing some icons. In that case, try to precompile the assets again. + +For Omnibus-packages: +``` +sudo gitlab-rake assets:precompile +``` + +For installations from source: +``` +cd /home/git/gitlab +sudo -u git -H bundle exec rake assets:precompile RAILS_ENV=production +``` diff --git a/doc/raketasks/user_management.md b/doc/raketasks/user_management.md index 3c67753ad28..80b01ca4043 100644 --- a/doc/raketasks/user_management.md +++ b/doc/raketasks/user_management.md @@ -6,7 +6,7 @@ # omnibus-gitlab sudo gitlab-rake gitlab:import:user_to_projects[username@domain.tld] -# installation from source or cookbook +# installation from source bundle exec rake gitlab:import:user_to_projects[username@domain.tld] RAILS_ENV=production ``` @@ -20,7 +20,7 @@ Notes: # omnibus-gitlab sudo gitlab-rake gitlab:import:all_users_to_all_projects -# installation from source or cookbook +# installation from source bundle exec rake gitlab:import:all_users_to_all_projects RAILS_ENV=production ``` @@ -30,7 +30,7 @@ bundle exec rake gitlab:import:all_users_to_all_projects RAILS_ENV=production # omnibus-gitlab sudo gitlab-rake gitlab:import:user_to_groups[username@domain.tld] -# installation from source or cookbook +# installation from source bundle exec rake gitlab:import:user_to_groups[username@domain.tld] RAILS_ENV=production ``` @@ -44,6 +44,6 @@ Notes: # omnibus-gitlab sudo gitlab-rake gitlab:import:all_users_to_all_groups -# installation from source or cookbook +# installation from source bundle exec rake gitlab:import:all_users_to_all_groups RAILS_ENV=production ``` diff --git a/doc/raketasks/web_hooks.md b/doc/raketasks/web_hooks.md index e1a58835d88..5a8b94af9b4 100644 --- a/doc/raketasks/web_hooks.md +++ b/doc/raketasks/web_hooks.md @@ -4,42 +4,42 @@ # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" - # source installations or cookbook + # source installations bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" RAILS_ENV=production ## Add a web hook for projects in a given **NAMESPACE**: # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme - # source installations or cookbook + # source installations bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production ## Remove a web hook from **ALL** projects using: # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" - # source installations or cookbook + # source installations bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" RAILS_ENV=production ## Remove a web hook from projects in a given **NAMESPACE**: # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme - # source installations or cookbook + # source installations bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production ## List **ALL** web hooks: # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:list - # source installations or cookbook + # source installations bundle exec rake gitlab:web_hook:list RAILS_ENV=production ## List the web hooks from projects in a given **NAMESPACE**: # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:list NAMESPACE=/ - # source installations or cookbook + # source installations bundle exec rake gitlab:web_hook:list NAMESPACE=/ RAILS_ENV=production > Note: `/` is the global namespace. diff --git a/doc/release/howto_rc1.md b/doc/release/howto_rc1.md new file mode 100644 index 00000000000..c4156d25d5f --- /dev/null +++ b/doc/release/howto_rc1.md @@ -0,0 +1,74 @@ +# How to create RC1 + +The RC1 release comes with the task to update the installation and upgrade docs. Be mindful that there might already be merge requests for this on GitLab or GitHub. + +### 1. Update the installation guide + +1. Check if it references the correct branch `x-x-stable` (doesn't exist yet, but that is okay) +1. Check the [GitLab Shell version](/lib/tasks/gitlab/check.rake#L782) +1. Check the [Git version](/lib/tasks/gitlab/check.rake#L794) +1. There might be other changes. Ask around. + +### 2. Create update guides + +[Follow this guide](howto_update_guides.md) to create update guides. + +### 3. Code quality indicators + +Make sure the code quality indicators are green / good. + +- [](http://ci.gitlab.org/projects/1?ref=master) on ci.gitlab.org (master branch) + +- [](https://semaphoreapp.com/gitlabhq/gitlabhq) (master branch) + +- [](https://codeclimate.com/github/gitlabhq/gitlabhq) + +- [](https://gemnasium.com/gitlabhq/gitlabhq) this button can be yellow (small updates are available) but must not be red (a security fix or an important update is available) + +- [](https://coveralls.io/r/gitlabhq/gitlabhq) + +### 4. Run release tool for CE and EE + +**Make sure EE `master` has latest changes from CE `master`** + +Get release tools + +``` +git clone git@dev.gitlab.org:gitlab/release-tools.git +cd release-tools +``` + +Release candidate creates stable branch from master. +So we need to sync master branch between all CE remotes. Also do same for EE. + +``` +bundle exec rake sync +``` + +Create release candidate and stable branch: + +``` +bundle exec rake release["x.x.0.rc1"] +``` + +Now developers can use master for merging new features. +So you should use stable branch for future code changes related to release. + + +### 5. Release GitLab CI RC1 + +Add to your local `gitlab-ci/.git/config`: + +``` +[remote "public"] + url = none + pushurl = git@dev.gitlab.org:gitlab/gitlab-ci.git + pushurl = git@gitlab.com:gitlab-org/gitlab-ci.git + pushurl = git@github.com:gitlabhq/gitlab-ci.git +``` + +* Create a stable branch `x-y-stable` +* Bump VERSION to `x.y.0.rc1` +* `git tag -a v$(cat VERSION) -m "Version $(cat VERSION)"` +* `git push public x-y-stable v$(cat VERSION)` + diff --git a/doc/release/howto_update_guides.md b/doc/release/howto_update_guides.md new file mode 100644 index 00000000000..23d0959c33d --- /dev/null +++ b/doc/release/howto_update_guides.md @@ -0,0 +1,55 @@ +# Create update guides + +1. Create: CE update guide from previous version. Like `7.3-to-7.4.md` +1. Create: CE to EE update guide in EE repository for latest version. +1. Update: `6.x-or-7.x-to-7.x.md` to latest version. +1. Create: CI update guide from previous version + +It's best to copy paste the previous guide and make changes where necessary. +The typical steps are listed below with any points you should specifically look at. + +#### 0. Any major changes? + +List any major changes here, so the user is aware of them before starting to upgrade. For instance: + +- Database updates +- Web server changes +- File structure changes + +#### 1. Stop server + +#### 2. Make backup + +#### 3. Do users need to update dependencies like `git`? + +- Check if the [GitLab Shell version](/lib/tasks/gitlab/check.rake#L782) changed since the last release. + +- Check if the [Git version](/lib/tasks/gitlab/check.rake#L794) changed since the last release. + +#### 4. Get latest code + +#### 5. Does GitLab shell need to be updated? + +#### 6. Install libs, migrations, etc. + +#### 7. Any config files updated since last release? + +Check if any of these changed since last release: + +- [lib/support/nginx/gitlab](/lib/support/nginx/gitlab) +- [lib/support/nginx/gitlab-ssl](/lib/support/nginx/gitlab-ssl) +- <https://gitlab.com/gitlab-org/gitlab-shell/commits/master/config.yml.example> +- [config/gitlab.yml.example](/config/gitlab.yml.example) +- [config/unicorn.rb.example](/config/unicorn.rb.example) +- [config/database.yml.mysql](/config/database.yml.mysql) +- [config/database.yml.postgresql](/config/database.yml.postgresql) +- [config/initializers/rack_attack.rb.example](/config/initializers/rack_attack.rb.example) +- [config/resque.yml.example](/config/resque.yml.example) + +#### 8. Need to update init script? + +Check if the `init.d/gitlab` script changed since last release: [lib/support/init.d/gitlab](/lib/support/init.d/gitlab) + +#### 9. Start application + +#### 10. Check application status diff --git a/doc/release/monthly.md b/doc/release/monthly.md index 386c19c0fe0..c9e6d3426bc 100644 --- a/doc/release/monthly.md +++ b/doc/release/monthly.md @@ -1,190 +1,116 @@ # Monthly Release -NOTE: This is a guide for GitLab developers. +NOTE: This is a guide used by the GitLab B.V. developers. -# **7 workdays before release - Code Freeze & Release Manager** +It starts 7 working days before the release. +The release manager doesn't have to perform all the work but must ensure someone is assigned. +The current release manager must schedule the appointment of the next release manager. +The new release manager should create overall issue to track the progress. -### **1. Stop merging in code, except for important bug fixes** +## Release Manager -### **2. Release Manager** +A release manager is selected that coordinates all releases the coming month, including the patch releases for previous releases. +The release manager has to make sure all the steps below are done and delegated where necessary. +This person should also make sure this document is kept up to date and issues are created and updated. -A release manager is selected that coordinates the entire release of this version. The release manager has to make sure all the steps below are done and delegated where necessary. This person should also make sure this document is kept up to date and issues are created and updated. +## Take vacations into account -### **3. Create an overall issue** +The time is measured in weekdays to compensate for weekends. +Do everything on time to prevent problems due to rush jobs or too little testing time. +Make sure that you take into account any vacations of maintainers. +If the release is falling behind immediately warn the team. + +## Create an overall issue and follow it Create issue for GitLab CE project(internal). Name it "Release x.x.x" for easier searching. Replace the dates with actual dates based on the number of workdays before the release. +All steps from issue template are explained below ``` -Xth: - -* Update the changelog (#LINK) -* Triage the omnibus-gitlab milestone - -Xth: - -* Merge CE in to EE (#LINK) -* Close the omnibus-gitlab milestone - -Xth: - -* Create x.x.0.rc1 (#LINK) -* Build package for GitLab.com (https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/administration.md#build-a-package) - -Xth: - -* Update GitLab.com with rc1 (#LINK) (https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/administration.md#deploy-the-package) -* Regression issue and tweet about rc1 (#LINK) -* Start blog post (#LINK) - -Xth: - -* Do QA and fix anything coming out of it (#LINK) - -22nd: - -* Release CE and EE (#LINK) - -Xth: - -* * Deploy to GitLab.com (#LINK) - -``` - -### **4. Update changelog** - -Any changes not yet added to the changelog are added by lead developer and in that merge request the complete team is asked if there is anything missing. - -### **5. Take weekend and vacations into account** - -Ensure that there is enough time to incorporate the findings of the release candidate, etc. - -# **6 workdays before release- Merge the CE into EE** - -Do this via a merge request. +Xth: (7 working days before the 22nd) -# **5 workdays before release - Create RC1** +- [ ] Code freeze +- [ ] Update the CE changelog (#LINK) +- [ ] Update the EE changelog (#LINK) +- [ ] Update the CI changelog (#LINK) +- [ ] Triage the omnibus-gitlab milestone -The RC1 release comes with the task to update the installation and upgrade docs. Be mindful that there might already be merge requests for this on GitLab or GitHub. +Xth: (6 working days before the 22nd) -### **1. Update the installation guide** +- [ ] Merge CE master in to EE master via merge request (#LINK) +- [ ] Create CE, EE, CI RC1 versions (#LINK) +- [ ] Determine QA person and notify this person -1. Check if it references the correct branch `x-x-stable` (doesn't exist yet, but that is okay) -1. Check the [GitLab Shell version](/lib/tasks/gitlab/check.rake#L782) -1. Check the [Git version](/lib/tasks/gitlab/check.rake#L794) -1. There might be other changes. Ask around. +Xth: (5 working days before the 22nd) -### **2. Create update guides** +- [ ] Do QA and fix anything coming out of it (#LINK) +- [ ] Close the omnibus-gitlab milestone -1. Create: CE update guide from previous version. Like `7.3-to-7.4.md` -1. Create: CE to EE update guide in EE repository for latest version. -1. Update: `6.x-or-7.x-to-7.x.md` to latest version. +Xth: (4 working days before the 22nd) -It's best to copy paste the previous guide and make changes where necessary. -The typical steps are listed below with any points you should specifically look at. +- [ ] Build rc1 package for GitLab.com (https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/administration.md#build-a-package) +- [ ] Update GitLab.com with rc1 (#LINK) (https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/administration.md#deploy-the-package) -#### 0. Any major changes? +Xth: (3 working days before the 22nd) -List any major changes here, so the user is aware of them before starting to upgrade. For instance: +- [ ] Create regression issues (CE, CI) (#LINK) +- [ ] Tweet about rc1 (#LINK) +- [ ] Prepare the blog post (#LINK) -- Database updates -- Web server changes -- File structure changes +Xth: (2 working days before the 22nd) -#### 1. Make backup +- [ ] Merge CE stable branch into EE stable branch +- [ ] Check that everyone is mentioned on the blog post (the reviewer should have done this one working day ago) -#### 2. Stop server +Xth: (1 working day before the 22nd) -#### 3. Do users need to update dependencies like `git`? +- [ ] Create CE, EE, CI stable versions (#LINK) +- [ ] Create Omnibus tags and build packages +- [ ] Update GitLab.com with the stable version (#LINK) -- Check if the [GitLab Shell version](/lib/tasks/gitlab/check.rake#L782) changed since the last release. - -- Check if the [Git version](/lib/tasks/gitlab/check.rake#L794) changed since the last release. - -#### 4. Get latest code - -#### 5. Does GitLab shell need to be updated? - -#### 6. Install libs, migrations, etc. - -#### 7. Any config files updated since last release? - -Check if any of these changed since last release: - -- [lib/support/nginx/gitlab](/lib/support/nginx/gitlab) -- [lib/support/nginx/gitlab-ssl](/lib/support/nginx/gitlab-ssl) -- <https://gitlab.com/gitlab-org/gitlab-shell/commits/master/config.yml.example> -- [config/gitlab.yml.example](/config/gitlab.yml.example) -- [config/unicorn.rb.example](/config/unicorn.rb.example) -- [config/database.yml.mysql](/config/database.yml.mysql) -- [config/database.yml.postgresql](/config/database.yml.postgresql) -- [config/initializers/rack_attack.rb.example](/config/initializers/rack_attack.rb.example) -- [config/resque.yml.example](/config/resque.yml.example) - -#### 8. Need to update init script? - -Check if the `init.d/gitlab` script changed since last release: [lib/support/init.d/gitlab](/lib/support/init.d/gitlab) - -#### 9. Start application - -#### 10. Check application status +22nd: -### **3. Code quality indicators** +- [ ] Release CE, EE and CI (#LINK) -Make sure the code quality indicators are green / good. +``` -- [](http://ci.gitlab.org/projects/1?ref=master) on ci.gitlab.org (master branch) +- - - -- [](https://semaphoreapp.com/gitlabhq/gitlabhq) (master branch) +## Code Freeze -- [](https://codeclimate.com/github/gitlabhq/gitlabhq) +Stop merging code in master, except for important bug fixes -- [](https://gemnasium.com/gitlabhq/gitlabhq) this button can be yellow (small updates are available) but must not be red (a security fix or an important update is available) +## Update changelog -- [](https://coveralls.io/r/gitlabhq/gitlabhq) +Any changes not yet added to the changelog are added by lead developer and in that merge request the complete team is +asked if there is anything missing. -### **4. Set VERSION** +There are three changelogs that need to be updated: CE, EE and CI. -Change version in VERSION to `x.x.0.rc1`. +## Create RC1 (CE, EE, CI) -### **5. Tag** +[Follow this How-to guide](howto_rc1.md) to create RC1. -Create an annotated tag that points to the version change commit: +## Prepare CHANGELOG for next release -``` -git tag -a vx.x.0.rc1 -m 'Version x.x.0.rc1' -``` +Once the stable branches have been created, update the CHANGELOG in `master` with the upcoming version. -Tags should be created for both GitLab CE and GitLab EE. Don't forget to push tags to all remotes. +## QA -``` -git push remote_name vx.x.0.rc1 -``` - -### **6. Create stable branches** - -For GitLab EE, append `-ee` to the branch. - -`x-x-stable-ee` - -``` -git checkout master -git pull -git checkout -b x-x-stable -git push <remote> x-x-stable -``` +Create issue on dev.gitlab.org `gitlab` repository, named "GitLab X.X QA" in order to keep track of the progress. -Now developers can use master for merging new features. -So you should use stable branch for future code chages related to release. +Use the omnibus packages of Enterprise Edition using [this guide](https://dev.gitlab.org/gitlab/gitlab-ee/blob/master/doc/release/manual_testing.md). +**NOTE** Upgrader can only be tested when tags are pushed to all repositories. Do not forget to confirm it is working before releasing. Note that in the issue. -# **4 workdays before release - Release RC1** +#### Fix anything coming out of the QA -### **1. Determine QA person +Create an issue with description of a problem, if it is quick fix fix it yourself otherwise contact the team for advice. -Notify person of QA day. +**NOTE** If there is a problem that cannot be fixed in a timely manner, reverting the feature is an option! If the feature is reverted, +create an issue about it in order to discuss the next steps after the release. -### **2. Update GitLab.com** +## Update GitLab.com with RC1 Merge the RC1 EE code into GitLab.com. Once the build is green, create a package. @@ -192,20 +118,7 @@ If there are big database migrations consider testing them with the production d Try to deploy in the morning. It is important to do this as soon as possible, so we can catch any errors before we release the full version. -### **3. Prepare the blog post** - -- Start with a complete copy of the [release blog template](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/doc/release_blog_template.md) and fill it out. -- Check the changelog of CE and EE for important changes. -- Create a WIP MR for the blog post -- Ask Dmitriy to add screenshots to the WIP MR. -- Decide with team who will be the MVP user. -- Create WIP MR for adding MVP to MVP page on website -- Add a note if there are security fixes: This release fixes an important security issue and we advise everyone to upgrade as soon as possible. -- Create a merge request on [GitLab.com](https://gitlab.com/gitlab-com/www-gitlab-com/tree/master) -- Assign to one reviewer who will fix spelling issues by editing the branch (can use the online editor) -- After the reviewer is finished the whole team will be mentioned to give their suggestions via line comments - -### **4. Create a regressions issue** +## Create a regressions issue On [the GitLab CE issue tracker on GitLab.com](https://gitlab.com/gitlab-org/gitlab-ce/issues/) create an issue titled "GitLab X.X regressions" add the following text: @@ -216,117 +129,78 @@ The release manager will comment here about the plans for patch releases. Assign the issue to the release manager and /cc all the core-team members active on the issue tracker. If there are any known bugs in the release add them immediately. -### **4. Tweet** +## Tweet about RC1 Tweet about the RC release: > GitLab x.x.0.rc1 is out. This release candidate is only suitable for testing. Please link regressions issues from LINK_TO_REGRESSION_ISSUE -# **1 workdays before release - Preparation** - -### **1. Pre QA merge** - -Merge CE into EE before doing the QA. - -### **2. QA** +## Prepare the blog post -Create issue on dev.gitlab.org `gitlab` repository, named "GitLab X.X QA" in order to keep track of the progress. - -Use the omnibus packages of Enterprise Edition using [this guide](https://dev.gitlab.org/gitlab/gitlab-ee/blob/master/doc/release/manual_testing.md). - -**NOTE** Upgrader can only be tested when tags are pushed to all repositories. Do not forget to confirm it is working before releasing. Note that in the issue. - -### **3. Fix anything coming out of the QA** - -Create an issue with description of a problem, if it is quick fix fix it yourself otherwise contact the team for advice. - -**NOTE** If there is a problem that cannot be fixed in a timely manner, reverting the feature is an option! If the feature is reverted, -create an issue about it in order to discuss the next steps after the release. +1. Start with a complete copy of the [release blog template](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/doc/release_blog_template.md) and fill it out. +1. Make sure the blog post contains information about the GitLab CI release. +1. Check the changelog of CE and EE for important changes. +1. Also check the CI changelog +1. Add a proposed tweet text to the blog post WIP MR description. +1. Create a WIP MR for the blog post +1. Ask Dmitriy to add screenshots to the WIP MR. +1. Decide with team who will be the MVP user. +1. Create WIP MR for adding MVP to MVP page on website +1. Add a note if there are security fixes: This release fixes an important security issue and we advise everyone to upgrade as soon as possible. +1. Create a merge request on [GitLab.com](https://gitlab.com/gitlab-com/www-gitlab-com/tree/master) +1. Assign to one reviewer who will fix spelling issues by editing the branch (either with a git client or by using the online editor) +1. Comment to the reviewer: '@person Please mention the whole team as soon as you are done (3 workdays before release at the latest)' -# **22nd - Release CE and EE** +## Create CE, EE, CI stable versions -For GitLab EE, append `-ee` to the branches and tags. +Get release tools -`x-x-stable-ee` - -`v.x.x.0-ee` +``` +git clone git@dev.gitlab.org:gitlab/release-tools.git +cd release-tools +``` -Note: Merge CE into EE if needed. +Bump version, create release tag and push to remotes: -### **1. Set VERSION to x.x.x and push** +``` +bundle exec rake release["x.x.0"] +``` -- Change the GITLAB_SHELL_VERSION file in `master` of the CE repository if the version changed. -- Change the GITLAB_SHELL_VERSION file in `master` of the EE repository if the version changed. -- Change the VERSION file in `master` branch of the CE repository and commit and push to origin. -- Change the VERSION file in `master` branch of the EE repository and commit and push to origin. +Also perform these steps for GitLab CI: -### **2. Update installation.md** +1. bump version in the stable branch +1. create annotated tag +1. push the stable branch and the annotated tag to the public repositories Update [installation.md](/doc/install/installation.md) to the newest version in master. -### **3. Push latest changes from x-x-stable branch to dev.gitlab.org** -``` -git checkout -b x-x-stable -git push origin x-x-stable -``` - -### **4. Build the Omnibus packages** +## Create Omnibus tags and build packages Follow the [release doc in the Omnibus repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/release.md). This can happen before tagging because Omnibus uses tags in its own repo and SHA1's to refer to the GitLab codebase. -### **5. Create annotated tag vx.x.x** - -In `x-x-stable` branch check for the SHA-1 of the commit with VERSION file changed. Tag that commit, - -``` -git tag -a vx.x.0 -m 'Version x.x.0' xxxxx -``` - -where `xxxxx` is SHA-1. - -### **6. Push the tag and x-x-stable branch to the remotes** -For GitLab CE, push to dev, GitLab.com and GitHub. +## Release CE, EE and CI -For GitLab EE, push to the subscribers repo. - -Make sure the branch is marked 'protected' on each of the remotes you pushed to. - -``` -git push <remote> x-x-stable(-ee) -git push <remote> vx.x.0 -``` - -### **7. Publish packages for new release** +__1. Publish packages for new release__ Update `downloads/index.html` and `downloads/archive/index.html` in `www-gitlab-com` repository. -### **8. Publish blog for new release** +__2. Publish blog for new release__ +Doublecheck the everyone has been mentioned in the blog post. Merge the [blog merge request](#1-prepare-the-blog-post) in `www-gitlab-com` repository. -### **9. Tweet to blog** +__3. Tweet to blog__ Send out a tweet to share the good news with the world. List the most important features and link to the blog post. -Proposed tweet for CE "GitLab X.X is released! It brings *** <link-to-blogpost>" - -# **1 workday after release - Update GitLab.com** - -Update GitLab.com from RC1 to the released package. - -# **25th - Release GitLab CI** +Proposed tweet "Release of GitLab X.X & CI Y.Y! FEATURE, FEATURE and FEATURE <link-to-blog-post> #gitlab" -- Create the update guid `doc/x.x-to-x.x.md`. -- Update CHANGELOG -- Bump version -- Create annotated tags `git tag -a vx.x.0 -m 'Version x.x.0' xxxxx` -- Create stable branch `x-x-stable` -- Create GitHub release post -- Post to blog about release -- Post to twitter +Consider creating a post on Hacker News. +## Update GitLab.com with the stable version +- Deploy the package (should not need downtime because of the small difference with RC1) diff --git a/doc/release/patch.md b/doc/release/patch.md index 5d2fa053cac..d8bb4aef0e2 100644 --- a/doc/release/patch.md +++ b/doc/release/patch.md @@ -10,22 +10,47 @@ Otherwise include it in the monthly release and note there was a regression fix ## Release Procedure +### Preparation + 1. Verify that the issue can be reproduced 1. Note in the 'GitLab X.X regressions' that you will create a patch 1. Create an issue on private GitLab development server 1. Name the issue "Release X.X.X CE and X.X.X EE", this will make searching easier 1. Fix the issue on a feature branch, do this on the private GitLab development server +1. If it is a security issue, then assign it to the release manager and apply a 'security' label 1. Consider creating and testing workarounds 1. After the branch is merged into master, cherry pick the commit(s) into the current stable branch -1. In a separate commit in the stable branch update the CHANGELOG -1. For EE, update the CHANGELOG-EE if it is EE specific fix. Otherwise, merge the stable CE branch and add to CHANGELOG-EE "Merge community edition changes for version X.X.X" -1. In a separate commit in the stable branch update the VERSION -1. Create an annotated tag vX.X.X for CE and another patch release for EE `git tag -a vx.x.x -m 'Version x.x.x'` 1. Make sure that the build has passed and all tests are passing -1. Push the code and the tags to all the CE and EE repositories -1. Apply the patch to GitLab Cloud and the private GitLab development server +1. In a separate commit in the master branch update the CHANGELOG +1. For EE, update the CHANGELOG-EE if it is EE specific fix. Otherwise, merge the stable CE branch and add to CHANGELOG-EE "Merge community edition changes for version X.X.X" +1. Merge CE stable branch into EE stable branch + + +### Bump version + +Get release tools + +``` +git clone git@dev.gitlab.org:gitlab/release-tools.git +cd release-tools +``` + +Bump version in stable branch, create release tag and push to remotes: + +``` +bundle exec rake release["x.x.x"] +``` + +Or if you need to release only EE: + +``` +CE=false be rake release['x.x.x'] +``` + +### Release + 1. [Build new packages with the latest version](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/release.md) -1. Cherry-pick the changelog update back into master +1. Apply the patch to GitLab.com and the private GitLab development server 1. Create and publish a blog post -1. Send tweets about the release from `@gitlabhq`, tweet should include the most important feature that the release is addressing and link to the blog post +1. Send tweets about the release from `@gitlab`, tweet should include the most important feature that the release is addressing and link to the blog post 1. Note in the 'GitLab X.X regressions' issue that the patch was published (CE only) diff --git a/doc/release/security.md b/doc/release/security.md index 79d23c02ea4..b67e0f37a04 100644 --- a/doc/release/security.md +++ b/doc/release/security.md @@ -14,11 +14,13 @@ Please report suspected security vulnerabilities in private to <support@gitlab.c 1. Verify that the issue can be reproduced 1. Acknowledge the issue to the researcher that disclosed it +1. Inform the release manager that there needs to be a security release 1. Do the steps from [patch release document](doc/release/patch.md), starting with "Create an issue on private GitLab development server" +1. The MR with the security fix should get a 'security' label and be assigned to the release manager +1. Build the package for GitLab.com and do a deploy 1. Create feature branches for the blog post on GitLab.com and link them from the code branch 1. Merge and publish the blog posts 1. Send tweets about the release from `@gitlabhq` -1. Send out an email to the 'GitLab Newsletter' mailing list on MailChimp (or the 'Subscribers' list if the security fix is for EE only) 1. Send out an email to [the community google mailing list](https://groups.google.com/forum/#!forum/gitlabhq) 1. Post a signed copy of our complete announcement to [oss-security](http://www.openwall.com/lists/oss-security/) and request a CVE number 1. Add the security researcher to the [Security Researcher Acknowledgments list](http://about.gitlab.com/vulnerability-acknowledgements/) diff --git a/doc/security/README.md b/doc/security/README.md index f88375f2afd..49dfa6eec76 100644 --- a/doc/security/README.md +++ b/doc/security/README.md @@ -2,4 +2,5 @@ - [Password length limits](password_length_limits.md) - [Rack attack](rack_attack.md) +- [Web Hooks and insecure internal web services](webhooks.md) - [Information exclusivity](information_exclusivity.md) diff --git a/doc/security/information_exclusivity.md b/doc/security/information_exclusivity.md index 127166ae2e7..f8e7fc3fd0e 100644 --- a/doc/security/information_exclusivity.md +++ b/doc/security/information_exclusivity.md @@ -4,6 +4,6 @@ Git is a distributed version control system (DVCS). This means that everyone that works with the source code has a local copy of the complete repository. In GitLab every project member that is not a guest (so reporters, developers and masters) can clone the repository to get a local copy. After obtaining this local copy the user can upload the full repository anywhere, including another project under their control or another server. -The consequense is that you can't build access controls that prevent the intentional sharing of source code by users that have access to the source code. +The consequence is that you can't build access controls that prevent the intentional sharing of source code by users that have access to the source code. This is an inherent feature of a DVCS and all git management systems have this limitation. Obviously you can take steps to prevent unintentional sharing and information destruction, this is why only some people are allowed to invite others and nobody can force push a protected branch. diff --git a/doc/security/webhooks.md b/doc/security/webhooks.md new file mode 100644 index 00000000000..1e9d33e87c3 --- /dev/null +++ b/doc/security/webhooks.md @@ -0,0 +1,13 @@ +# Web Hooks and insecure internal web services + +If you have non-GitLab web services running on your GitLab server or within its local network, these may be vulnerable to exploitation via Web Hooks. + +With [Web Hooks](../web_hooks/web_hooks.md), you and your project masters and owners can set up URLs to be triggered when specific things happen to projects. Normally, these requests are sent to external web services specifically set up for this purpose, that process the request and its attached data in some appropriate way. + +Things get hairy, however, when a Web Hook is set up with a URL that doesn't point to an external, but to an internal service, that may do something completely unintended when the web hook is triggered and the POST request is sent. + +Because Web Hook requests are made by the GitLab server itself, these have complete access to everything running on the server (http://localhost:123) or within the server's local network (http://192.168.1.12:345), even if these services are otherwise protected and inaccessible from the outside world. + +If a web service does not require authentication, Web Hooks can be used to trigger destructive commands by getting the GitLab server to make POST requests to endpoints like "http://localhost:123/some-resource/delete". + +To prevent this type of exploitation from happening, make sure that you are aware of every web service GitLab could potentially have access to, and that all of these are set up to require authentication for every potentially destructive command. Enabling authentication but leaving a default password is not enough.
\ No newline at end of file diff --git a/doc/ssh/README.md b/doc/ssh/README.md index c87fffd7d2c..6fe23dfa2a6 100644 --- a/doc/ssh/README.md +++ b/doc/ssh/README.md @@ -1,4 +1,73 @@ # SSH -- [Deploy keys](deploy_keys.md) -- [SSH](ssh.md) +## SSH keys + +An SSH key allows you to establish a secure connection between your +computer and GitLab. + +Before generating an SSH key, check if your system already has one by +running `cat ~/.ssh/id_rsa.pub`. If you see a long string starting with +`ssh-rsa` or `ssh-dsa`, you can skip the ssh-keygen step. + +To generate a new SSH key, just open your terminal and use code below. The +ssh-keygen command prompts you for a location and filename to store the key +pair and for a password. When prompted for the location and filename, you +can press enter to use the default. + +It is a best practice to use a password for an SSH key, but it is not +required and you can skip creating a password by pressing enter. Note that +the password you choose here can't be altered or retrieved. + +```bash +ssh-keygen -t rsa -C "$your_email" +``` + +Use the code below to show your public key. + +```bash +cat ~/.ssh/id_rsa.pub +``` + +Copy-paste the key to the 'My SSH Keys' section under the 'SSH' tab in your +user profile. Please copy the complete key starting with `ssh-` and ending +with your username and host. + +Use code below to copy your public key to the clipboard. Depending on your +OS you'll need to use a different command: + +**Windows:** +```bash +clip < ~/.ssh/id_rsa.pub +``` + +**Mac:** +```bash +pbcopy < ~/.ssh/id_rsa.pub +``` + +**Linux (requires xclip):** +```bash +xclip -sel clip < ~/.ssh/id_rsa.pub +``` + +## Deploy keys + +Deploy keys allow read-only access to multiple projects with a single SSH +key. + +This is really useful for cloning repositories to your Continuous +Integration (CI) server. By using deploy keys, you don't have to setup a +dummy user account. + +If you are a project master or owner, you can add a deploy key in the +project settings under the section 'Deploy Keys'. Press the 'New Deploy +Key' button and upload a public SSH key. After this, the machine that uses +the corresponding private key has read-only access to the project. + +You can't add the same deploy key twice with the 'New Deploy Key' option. +If you want to add the same key to another project, please enable it in the +list that says 'Deploy keys from projects available to you'. All the deploy +keys of all the projects you have access to are available. This project +access can happen through being a direct member of the projecti, or through +a group. See `def accessible_deploy_keys` in `app/models/user.rb` for more +information. diff --git a/doc/ssh/deploy_keys.md b/doc/ssh/deploy_keys.md deleted file mode 100644 index dcca8bdc61a..00000000000 --- a/doc/ssh/deploy_keys.md +++ /dev/null @@ -1,9 +0,0 @@ -# Deploy keys - -Deploy keys allow read-only access one or multiple projects with a single SSH key. - -This is really useful for cloning repositories to your Continuous Integration (CI) server. By using a deploy keys you don't have to setup a dummy user account. - -If you are a project master or owner you can add a deploy key in the project settings under the section Deploy Keys. Press the 'New Deploy Key' button and upload a public ssh key. After this the machine that uses the corresponding private key has read-only access to the project. - -You can't add the same deploy key twice with the 'New Deploy Key' option. If you want to add the same key to another project please enable it in the list that says 'Deploy keys from projects available to you'. All the deploy keys of all the projects you have access to are available. This project access can happen through being a direct member of the project or through a group. See `def accessible_deploy_keys` in `app/models/user.rb` for more information. diff --git a/doc/ssh/ssh.md b/doc/ssh/ssh.md deleted file mode 100644 index d466c1bde72..00000000000 --- a/doc/ssh/ssh.md +++ /dev/null @@ -1,21 +0,0 @@ -# SSH keys - -SSH key allows you to establish a secure connection between your computer and GitLab - -Before generating an SSH key, check if your system already has one by running `cat ~/.ssh/id_rsa.pub` If your see a long string starting with `ssh-rsa` or `ssh-dsa`, you can skip the ssh-keygen step. - -To generate a new SSH key just open your terminal and use code below. The ssh-keygen command prompts you for a location and filename to store the key pair and for a password. When prompted for the location and filename you can press enter to use the default. -It is a best practice to use a password for an SSH key but it is not required and you can skip creating a password by pressing enter. -Note that the password you choose here can't be altered or retrieved. - -```bash -ssh-keygen -t rsa -C "$your_email" -``` - -Use the code below to show your public key. - -```bash -cat ~/.ssh/id_rsa.pub -``` - -Copy-paste the key to the 'My SSH Keys' section under the 'SSH' tab in your user profile. Please copy the complete key starting with `ssh-` and ending with your username and host. diff --git a/doc/system_hooks/system_hooks.md b/doc/system_hooks/system_hooks.md index 54e6e3a9e3f..f9b6d37d840 100644 --- a/doc/system_hooks/system_hooks.md +++ b/doc/system_hooks/system_hooks.md @@ -1,6 +1,6 @@ # System hooks -Your GitLab instance can perform HTTP POST requests on the following events: `project_create`, `project_destroy`, `user_add_to_team`, `user_remove_from_team`, `user_create`, `user_destroy`, `key_create` and `key_destroy`. +Your GitLab instance can perform HTTP POST requests on the following events: `project_create`, `project_destroy`, `user_add_to_team`, `user_remove_from_team`, `user_create`, `user_destroy`, `key_create`, `key_destroy`, `group_create`, `group_destroy`, `user_add_to_group` and `user_remove_from_group`. System hooks can be used, e.g. for logging or changing information in a LDAP server. @@ -15,8 +15,8 @@ System hooks can be used, e.g. for logging or changing information in a LDAP ser "name": "StoreCloud", "owner_email": "johnsmith@gmail.com", "owner_name": "John Smith", - "path": "stormcloud", - "path_with_namespace": "jsmith/stormcloud", + "path": "storecloud", + "path_with_namespace": "jsmith/storecloud", "project_id": 74, "project_visibility": "private", } @@ -50,6 +50,7 @@ System hooks can be used, e.g. for logging or changing information in a LDAP ser "project_path": "storecloud", "user_email": "johnsmith@gmail.com", "user_name": "John Smith", + "user_id": 41, "project_visibility": "private", } ``` @@ -66,6 +67,7 @@ System hooks can be used, e.g. for logging or changing information in a LDAP ser "project_path": "storecloud", "user_email": "johnsmith@gmail.com", "user_name": "John Smith", + "user_id": 41, "project_visibility": "private", } ``` @@ -117,3 +119,62 @@ System hooks can be used, e.g. for logging or changing information in a LDAP ser "id": 4 } ``` + +**Group created:** + +```json +{ + "created_at": "2012-07-21T07:30:54Z", + "event_name": "group_create", + "name": "StoreCloud", + "owner_email": "johnsmith@gmail.com", + "owner_name": "John Smith", + "path": "storecloud", + "group_id": 78 +} +``` + +**Group removed:** + +```json +{ + "created_at": "2012-07-21T07:30:54Z", + "event_name": "group_destroy", + "name": "StoreCloud", + "owner_email": "johnsmith@gmail.com", + "owner_name": "John Smith", + "path": "storecloud", + "group_id": 78 +} +``` + +**New Group Member:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "event_name": "user_add_to_group", + "group_access": "Master", + "group_id": 78, + "group_name": "StoreCloud", + "group_path": "storecloud", + "user_email": "johnsmith@gmail.com", + "user_name": "John Smith", + "user_id": 41 +} +``` +**Group Member Removed:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "event_name": "user_remove_from_group", + "group_access": "Master", + "group_id": 78, + "group_name": "StoreCloud", + "group_path": "storecloud", + "user_email": "johnsmith@gmail.com", + "user_name": "John Smith", + "user_id": 41 +} +``` diff --git a/doc/update/2.6-to-3.0.md b/doc/update/2.6-to-3.0.md index 6aabbe095dc..4827ef9501a 100644 --- a/doc/update/2.6-to-3.0.md +++ b/doc/update/2.6-to-3.0.md @@ -1,4 +1,5 @@ # From 2.6 to 3.0 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/2.6-to-3.0.md) for the most up to date instructions.* ## 1. Stop server & resque @@ -22,29 +23,29 @@ sudo -u gitlab bundle exec rake db:migrate RAILS_ENV=production # !!! Config should be replaced with a new one. Check it after replace cp config/gitlab.yml.example config/gitlab.yml -# update gitolite hooks +# update Gitolite hooks -# GITOLITE v2: +# Gitolite v2: sudo cp ./lib/hooks/post-receive /home/git/share/gitolite/hooks/common/post-receive sudo chown git:git /home/git/share/gitolite/hooks/common/post-receive -# GITOLITE v3: +# Gitolite v3: sudo cp ./lib/hooks/post-receive /home/git/.gitolite/hooks/common/post-receive sudo chown git:git /home/git/.gitolite/hooks/common/post-receive # set valid path to hooks in gitlab.yml in git_host section # like this git_host: - # gitolite 2 + # Gitolite 2 hooks_path: /home/git/share/gitolite/hooks - # gitolite 3 + # Gitolite 3 hooks_path: /home/git/.gitolite/hooks/ -# Make some changes to gitolite config +# Make some changes to Gitolite config # For more information visit https://github.com/gitlabhq/gitlabhq/pull/1719 -# gitolite v2 +# Gitolite v2 sudo -u git -H sed -i 's/\(GL_GITCONFIG_KEYS\s*=>*\s*\).\{2\}/\\1"\.\*"/g' /home/git/.gitolite.rc # gitlite v3 diff --git a/doc/update/2.9-to-3.0.md b/doc/update/2.9-to-3.0.md index 8af86b0dc98..f4a997a8c5e 100644 --- a/doc/update/2.9-to-3.0.md +++ b/doc/update/2.9-to-3.0.md @@ -1,4 +1,5 @@ # From 2.9 to 3.0 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/2.9-to-3.0.md) for the most up to date instructions.* ## 1. Stop server & resque diff --git a/doc/update/3.0-to-3.1.md b/doc/update/3.0-to-3.1.md index 3206df3499b..a30485c42f7 100644 --- a/doc/update/3.0-to-3.1.md +++ b/doc/update/3.0-to-3.1.md @@ -1,4 +1,5 @@ # From 3.0 to 3.1 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/3.0-to-3.1.md) for the most up to date instructions.* **IMPORTANT!** diff --git a/doc/update/3.1-to-4.0.md b/doc/update/3.1-to-4.0.md index 165f4e6a308..f1ef4df4744 100644 --- a/doc/update/3.1-to-4.0.md +++ b/doc/update/3.1-to-4.0.md @@ -1,4 +1,5 @@ # From 3.1 to 4.0 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/3.1-to-4.0.md) for the most up to date instructions.* ## Important changes diff --git a/doc/update/4.0-to-4.1.md b/doc/update/4.0-to-4.1.md index 4149ed6b08d..d89d5235917 100644 --- a/doc/update/4.0-to-4.1.md +++ b/doc/update/4.0-to-4.1.md @@ -1,4 +1,5 @@ # From 4.0 to 4.1 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/4.0-to-4.1.md) for the most up to date instructions.* ## Important changes diff --git a/doc/update/4.1-to-4.2.md b/doc/update/4.1-to-4.2.md index 5ee8e8781e9..6fe4412ff90 100644 --- a/doc/update/4.1-to-4.2.md +++ b/doc/update/4.1-to-4.2.md @@ -1,4 +1,5 @@ # From 4.1 to 4.2 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/4.1-to-4.2.md) for the most up to date instructions.* ## 1. Stop server & Resque diff --git a/doc/update/4.2-to-5.0.md b/doc/update/4.2-to-5.0.md index cde679598f7..f9faf65f952 100644 --- a/doc/update/4.2-to-5.0.md +++ b/doc/update/4.2-to-5.0.md @@ -1,4 +1,5 @@ # From 4.2 to 5.0 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/4.2-to-5.0.md) for the most up to date instructions.* ## Warning @@ -41,8 +42,8 @@ git checkout v1.1.0 # copy config cp config.yml.example config.yml -# change url to GitLab instance -# ! make sure url end with '/' like 'https://gitlab.example/' +# change URL to GitLab instance +# ! make sure the URL ends with '/' like 'https://gitlab.example/' vim config.yml # rewrite hooks @@ -111,7 +112,7 @@ sudo chmod -R u+rwX /home/git/gitlab/tmp/pids ``` -## 6. Update init.d script and nginx config +## 6. Update init.d script and Nginx config ```bash # init.d @@ -123,7 +124,7 @@ sudo chmod +x /etc/init.d/gitlab sudo -u git -H cp /home/git/gitlab/config/unicorn.rb /home/git/gitlab/config/unicorn.rb.old sudo -u git -H cp /home/git/gitlab/config/unicorn.rb.example /home/git/gitlab/config/unicorn.rb -#nginx +# Nginx # Replace path from '/home/gitlab/' to '/home/git/' sudo vim /etc/nginx/sites-enabled/gitlab sudo service nginx restart @@ -137,7 +138,7 @@ sudo service gitlab start # check if unicorn and sidekiq started # If not try to logout, also check replaced path from '/home/gitlab/' to '/home/git/' -# in nginx, unicorn, init.d etc +# in Nginx, unicorn, init.d etc ps aux | grep unicorn ps aux | grep sidekiq diff --git a/doc/update/5.0-to-5.1.md b/doc/update/5.0-to-5.1.md index 0e597abb1a9..9fbd1f88515 100644 --- a/doc/update/5.0-to-5.1.md +++ b/doc/update/5.0-to-5.1.md @@ -1,4 +1,5 @@ # From 5.0 to 5.1 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/5.0-to-5.1.md) for the most up to date instructions.* ## Warning diff --git a/doc/update/5.1-to-5.2.md b/doc/update/5.1-to-5.2.md index 6ef559ac9f9..cf9c4e4f770 100644 --- a/doc/update/5.1-to-5.2.md +++ b/doc/update/5.1-to-5.2.md @@ -1,4 +1,5 @@ # From 5.1 to 5.2 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/5.1-to-5.2.md) for the most up to date instructions.* ## Warning diff --git a/doc/update/5.1-to-5.4.md b/doc/update/5.1-to-5.4.md index 8ec56b266ca..97a98ede070 100644 --- a/doc/update/5.1-to-5.4.md +++ b/doc/update/5.1-to-5.4.md @@ -1,4 +1,5 @@ # From 5.1 to 5.4 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/5.1-to-5.4.md) for the most up to date instructions.* Also works starting from 5.2. diff --git a/doc/update/5.1-to-6.0.md b/doc/update/5.1-to-6.0.md index a76b371e6d6..a3fdd92bd2f 100644 --- a/doc/update/5.1-to-6.0.md +++ b/doc/update/5.1-to-6.0.md @@ -1,4 +1,5 @@ # From 5.1 to 6.0 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/5.1-to-6.0.md) for the most up to date instructions.* ## Warning @@ -40,7 +41,7 @@ sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production The migrations in this update are very sensitive to incomplete or inconsistent data. If you have a long-running GitLab installation and some of the previous upgrades did not work out 100% correct this may bite you now. The following can help you have a more smooth upgrade. -### Find projets with invalid project names +### Find projects with invalid project names #### MySQL Login to MySQL: diff --git a/doc/update/5.2-to-5.3.md b/doc/update/5.2-to-5.3.md index 61ddf135641..27613aeda07 100644 --- a/doc/update/5.2-to-5.3.md +++ b/doc/update/5.2-to-5.3.md @@ -1,4 +1,5 @@ # From 5.2 to 5.3 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/5.2-to-5.3.md) for the most up to date instructions.* ## Warning diff --git a/doc/update/5.3-to-5.4.md b/doc/update/5.3-to-5.4.md index 8a0d43e3e64..577b9a585ff 100644 --- a/doc/update/5.3-to-5.4.md +++ b/doc/update/5.3-to-5.4.md @@ -1,4 +1,5 @@ # From 5.3 to 5.4 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/5.3-to-5.4.md) for the most up to date instructions.* ## 0. Backup diff --git a/doc/update/5.4-to-6.0.md b/doc/update/5.4-to-6.0.md index 7bf7bce6aa0..d9c6d9bfb91 100644 --- a/doc/update/5.4-to-6.0.md +++ b/doc/update/5.4-to-6.0.md @@ -1,16 +1,20 @@ # From 5.4 to 6.0 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/5.4-to-6.0.md) for the most up to date instructions.* ## Warning GitLab 6.0 is affected by critical security vulnerabilities CVE-2013-4490 and CVE-2013-4489. +**You need to follow this guide first, before updating past 6.0, as it contains critical migration steps that are only present +in the `6-0-stable` branch** + ## Deprecations ### Global projects The root (global) namespace for projects is deprecated. -So you need to move all your global projects under groups or users manually before update or they will be automatically moved to the project owner namespace during the update. When a project is moved all its members will receive an email with instructions how to update their git remote url. Please make sure you disable sending email when you do a test of the upgrade. +So you need to move all your global projects under groups or users manually before update or they will be automatically moved to the project owner namespace during the update. When a project is moved all its members will receive an email with instructions how to update their git remote URL. Please make sure you disable sending email when you do a test of the upgrade. ### Teams diff --git a/doc/update/6.0-to-6.1.md b/doc/update/6.0-to-6.1.md index 9d67a3bcb96..c5eba1c01c4 100644 --- a/doc/update/6.0-to-6.1.md +++ b/doc/update/6.0-to-6.1.md @@ -1,4 +1,5 @@ # From 6.0 to 6.1 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.0-to-6.1.md) for the most up to date instructions.* ## Warning diff --git a/doc/update/6.1-to-6.2.md b/doc/update/6.1-to-6.2.md index efa6e43124c..a534528108a 100644 --- a/doc/update/6.1-to-6.2.md +++ b/doc/update/6.1-to-6.2.md @@ -1,4 +1,5 @@ # From 6.1 to 6.2 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.1-to-6.2.md) for the most up to date instructions.* **You should update to 6.1 before installing 6.2 so all the necessary conversions are run.** @@ -35,7 +36,7 @@ sudo -u git -H git checkout v1.7.9 # Addresses multiple critical security vulner ## 4. Install additional packages ```bash -# Add support for lograte for better log file handling +# Add support for logrotate for better log file handling sudo apt-get install logrotate ``` diff --git a/doc/update/6.2-to-6.3.md b/doc/update/6.2-to-6.3.md index e9b3bdd2f54..b08ebde0808 100644 --- a/doc/update/6.2-to-6.3.md +++ b/doc/update/6.2-to-6.3.md @@ -1,4 +1,5 @@ # From 6.2 to 6.3 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.2-to-6.3.md) for the most up to date instructions.* **Requires version: 6.1 or 6.2.** diff --git a/doc/update/6.3-to-6.4.md b/doc/update/6.3-to-6.4.md index 96c2895981d..951d92dfeb5 100644 --- a/doc/update/6.3-to-6.4.md +++ b/doc/update/6.3-to-6.4.md @@ -1,4 +1,5 @@ # From 6.3 to 6.4 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.3-to-6.4.md) for the most up to date instructions.* ## 0. Backup diff --git a/doc/update/6.4-to-6.5.md b/doc/update/6.4-to-6.5.md index 1624296fc3f..0dae9a9fe59 100644 --- a/doc/update/6.4-to-6.5.md +++ b/doc/update/6.4-to-6.5.md @@ -1,4 +1,5 @@ # From 6.4 to 6.5 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.4-to-6.5.md) for the most up to date instructions.* ## 0. Backup diff --git a/doc/update/6.5-to-6.6.md b/doc/update/6.5-to-6.6.md index 544eee17fec..c24e83eb006 100644 --- a/doc/update/6.5-to-6.6.md +++ b/doc/update/6.5-to-6.6.md @@ -1,4 +1,5 @@ # From 6.5 to 6.6 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.5-to-6.6.md) for the most up to date instructions.* ## 0. Backup diff --git a/doc/update/6.6-to-6.7.md b/doc/update/6.6-to-6.7.md index 77ac4d0bfa6..5622a7001ed 100644 --- a/doc/update/6.6-to-6.7.md +++ b/doc/update/6.6-to-6.7.md @@ -1,4 +1,5 @@ # From 6.6 to 6.7 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.6-to-6.7.md) for the most up to date instructions.* ## 0. Backup diff --git a/doc/update/6.7-to-6.8.md b/doc/update/6.7-to-6.8.md index 16f3439c998..4fb90639f16 100644 --- a/doc/update/6.7-to-6.8.md +++ b/doc/update/6.7-to-6.8.md @@ -1,4 +1,5 @@ # From 6.7 to 6.8 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.7-to-6.8.md) for the most up to date instructions.* ## 0. Backup diff --git a/doc/update/6.8-to-6.9.md b/doc/update/6.8-to-6.9.md index 9efb384ff59..b9b8b63f652 100644 --- a/doc/update/6.8-to-6.9.md +++ b/doc/update/6.8-to-6.9.md @@ -1,4 +1,5 @@ # From 6.8 to 6.9 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.8-to-6.9.md) for the most up to date instructions.* ### 0. Backup diff --git a/doc/update/6.9-to-7.0.md b/doc/update/6.9-to-7.0.md index 1f3421a799b..236430b5951 100644 --- a/doc/update/6.9-to-7.0.md +++ b/doc/update/6.9-to-7.0.md @@ -1,4 +1,5 @@ # From 6.9 to 7.0 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.9-to-7.0.md) for the most up to date instructions.* ### 0. Backup diff --git a/doc/update/6.x-or-7.x-to-7.4.md b/doc/update/6.x-or-7.x-to-7.8.md index dd90ae3bf3d..673d9253d62 100644 --- a/doc/update/6.x-or-7.x-to-7.4.md +++ b/doc/update/6.x-or-7.x-to-7.8.md @@ -1,6 +1,7 @@ -# From 6.x or 7.x to 7.4 +# From 6.x or 7.x to 7.8 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.x-or-7.x-to-7.8.md) for the most up to date instructions.* -This allows you to upgrade any version of GitLab from 6.0 and up (including 7.0 and up) to 7.4. +This allows you to upgrade any version of GitLab from 6.0 and up (including 7.0 and up) to 7.8. ## Global issue numbers @@ -34,7 +35,7 @@ You can check which version you are running with `ruby -v`. If you are you running Ruby 2.0.x, you do not need to upgrade ruby, but can consider doing so for performance reasons. -If you are running Ruby 2.1.1 consider upgrading to 2.1.2, because of the high memory usage of Ruby 2.1.1. +If you are running Ruby 2.1.1 consider upgrading to 2.1.5, because of the high memory usage of Ruby 2.1.1. Install, update dependencies: @@ -46,8 +47,8 @@ Download and compile Ruby: ```bash mkdir /tmp/ruby && cd /tmp/ruby -curl --progress ftp://ftp.ruby-lang.org/pub/ruby/2.1/ruby-2.1.2.tar.gz | tar xz -cd ruby-2.1.2 +curl --progress http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.5.tar.gz | tar xz +cd ruby-2.1.5 ./configure --disable-install-rdoc make sudo make install @@ -70,7 +71,7 @@ sudo -u git -H git checkout -- db/schema.rb # local changes will be restored aut For GitLab Community Edition: ```bash -sudo -u git -H git checkout 7-4-stable +sudo -u git -H git checkout 7-8-stable ``` OR @@ -78,17 +79,20 @@ OR For GitLab Enterprise Edition: ```bash -sudo -u git -H git checkout 7-4-stable-ee +sudo -u git -H git checkout 7-8-stable-ee ``` ## 4. Install additional packages ```bash -# Add support for lograte for better log file handling +# Add support for logrotate for better log file handling sudo apt-get install logrotate # Install pkg-config and cmake, which is needed for the latest versions of rugged sudo apt-get install pkg-config cmake + +# Install Kerberos header files, which are needed for GitLab EE Kerberos support +sudo apt-get install libkrb5-dev ``` ## 5. Configure Redis to use sockets @@ -119,7 +123,7 @@ sudo apt-get install pkg-config cmake ```bash cd /home/git/gitlab-shell sudo -u git -H git fetch -sudo -u git -H git checkout v2.0.1 +sudo -u git -H git checkout v2.5.4 ``` ## 7. Install libs, migrations, etc. @@ -154,14 +158,12 @@ sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab TIP: to see what changed in `gitlab.yml.example` in this release use next command: ``` -git diff 6-0-stable:config/gitlab.yml.example 7-4-stable:config/gitlab.yml.example +git diff 6-0-stable:config/gitlab.yml.example 7-8-stable:config/gitlab.yml.example ``` -* Make `/home/git/gitlab/config/gitlab.yml` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-4-stable/config/gitlab.yml.example but with your settings. -* Make `/home/git/gitlab/config/unicorn.rb` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-4-stable/config/unicorn.rb.example but with your settings. -* Make `/home/git/gitlab-shell/config.yml` the same as https://gitlab.com/gitlab-org/gitlab-shell/blob/v2.0.1/config.yml.example but with your settings. -* HTTP setups: Make `/etc/nginx/sites-available/gitlab` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-4-stable/lib/support/nginx/gitlab but with your settings. -* HTTPS setups: Make `/etc/nginx/sites-available/gitlab-ssl` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-4-stable/lib/support/nginx/gitlab-ssl but with your settings. +* Make `/home/git/gitlab/config/gitlab.yml` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-8-stable/config/gitlab.yml.example but with your settings. +* Make `/home/git/gitlab/config/unicorn.rb` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-8-stable/config/unicorn.rb.example but with your settings. +* Make `/home/git/gitlab-shell/config.yml` the same as https://gitlab.com/gitlab-org/gitlab-shell/blob/v2.5.4/config.yml.example but with your settings. * Copy rack attack middleware config ```bash @@ -174,6 +176,12 @@ sudo -u git -H cp config/initializers/rack_attack.rb.example config/initializers sudo cp lib/support/logrotate/gitlab /etc/logrotate.d/gitlab ``` +### Change Nginx settings + +* HTTP setups: Make `/etc/nginx/sites-available/gitlab` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-8-stable/lib/support/nginx/gitlab but with your settings. +* HTTPS setups: Make `/etc/nginx/sites-available/gitlab-ssl` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-8-stable/lib/support/nginx/gitlab-ssl but with your settings. +* A new `location /uploads/` section has been added that needs to have the same content as the existing `location @gitlab` section. + ## 9. Start application sudo service gitlab start @@ -196,7 +204,7 @@ If all items are green, then congratulations upgrade complete! When using Google omniauth login, changes of the Google account required. Ensure that `Contacts API` and the `Google+ API` are enabled in the [Google Developers Console](https://console.developers.google.com/). -More details can be found at the [integration documentation](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/integration/google.md). +More details can be found at the [integration documentation](../../../master/doc/integration/google.md). ## 12. Optional optimizations for GitLab setups with MySQL databases @@ -217,13 +225,13 @@ mysql -u root -p # Convert all tables to use the InnoDB storage engine (added in GitLab 6.8) SELECT CONCAT('ALTER TABLE gitlabhq_production.', table_name, ' ENGINE=InnoDB;') AS 'Copy & run these SQL statements:' FROM information_schema.tables WHERE table_schema = 'gitlabhq_production' AND `ENGINE` <> 'InnoDB' AND `TABLE_TYPE` = 'BASE TABLE'; -# If previous query returned results, copy & run all outputed SQL statements +# If previous query returned results, copy & run all shown SQL statements # Convert all tables to correct character set SET foreign_key_checks = 0; SELECT CONCAT('ALTER TABLE gitlabhq_production.', table_name, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;') AS 'Copy & run these SQL statements:' FROM information_schema.tables WHERE table_schema = 'gitlabhq_production' AND `TABLE_COLLATION` <> 'utf8_unicode_ci' AND `TABLE_TYPE` = 'BASE TABLE'; -# If previous query returned results, copy & run all outputed SQL statements +# If previous query returned results, copy & run all shown SQL statements # turn foreign key checks back on SET foreign_key_checks = 1; @@ -231,7 +239,7 @@ SET foreign_key_checks = 1; # Find MySQL users mysql> SELECT user FROM mysql.user WHERE user LIKE '%git%'; -# If git user exists and gitlab user does not exist +# If git user exists and gitlab user does not exist # you are done with the database cleanup tasks mysql> \q @@ -269,11 +277,11 @@ mysql> \q sudo -u git -H editor /home/git/gitlab/config/database.yml ``` -## Things went south? Revert to previous version (6.0) +## Things went south? Revert to previous version (7.0) ### 1. Revert the code to the previous version -Follow the [upgrade guide from 5.4 to 6.0](5.4-to-6.0.md), except for the database migration (the backup is already migrated to the previous version). +Follow the [upgrade guide from 6.9 to 7.0](6.9-to-7.0.md), except for the database migration (the backup is already migrated to the previous version). ### 2. Restore from the backup: diff --git a/doc/update/7.0-to-7.1.md b/doc/update/7.0-to-7.1.md index 82bb5708734..a4e9be9946e 100644 --- a/doc/update/7.0-to-7.1.md +++ b/doc/update/7.0-to-7.1.md @@ -1,4 +1,5 @@ # From 7.0 to 7.1 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/7.0-to-7.1.md) for the most up to date instructions.* ### 0. Backup diff --git a/doc/update/7.1-to-7.2.md b/doc/update/7.1-to-7.2.md index 699111f0143..88cb63d7d41 100644 --- a/doc/update/7.1-to-7.2.md +++ b/doc/update/7.1-to-7.2.md @@ -1,4 +1,5 @@ # From 7.1 to 7.2 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/7.1-to-7.2.md) for the most up to date instructions.* ## Editable labels diff --git a/doc/update/7.2-to-7.3.md b/doc/update/7.2-to-7.3.md index ebdd4ff60fa..18f77d6396e 100644 --- a/doc/update/7.2-to-7.3.md +++ b/doc/update/7.2-to-7.3.md @@ -1,4 +1,5 @@ # From 7.2 to 7.3 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/7.2-to-7.3.md) for the most up to date instructions.* ### 0. Backup diff --git a/doc/update/7.3-to-7.4.md b/doc/update/7.3-to-7.4.md index 3f471500c82..53e739c06fb 100644 --- a/doc/update/7.3-to-7.4.md +++ b/doc/update/7.3-to-7.4.md @@ -1,4 +1,5 @@ # From 7.3 to 7.4 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/7.3-to-7.4.md) for the most up to date instructions.* ### 0. Stop server @@ -35,8 +36,6 @@ sudo -u git -H git checkout 7-4-stable-ee ### 3. Install libs, migrations, etc. ```bash -cd /home/git/gitlab - # MySQL installations (note: the line below states '--without ... postgres') sudo -u git -H bundle install --without development test postgres --deployment @@ -70,14 +69,17 @@ git diff origin/7-3-stable:config/gitlab.yml.example origin/7-4-stable:config/gi sudo -u git -H editor config/unicorn.rb ``` -#### Change nginx https settings +#### Change Nginx HTTPS settings * HTTPS setups: Make `/etc/nginx/sites-available/gitlab-ssl` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-4-stable/lib/support/nginx/gitlab-ssl but with your setting #### MySQL Databases: Update database.yml config file -* Add `collation: utf8_general_ci` to config/database.yml as seen in [config/database.yml.mysql](config/database.yml.mysql) +* Add `collation: utf8_general_ci` to `config/database.yml` as seen in [config/database.yml.mysql](/config/database.yml.mysql) +``` +sudo -u git -H editor config/database.yml +``` ### 5. Start application @@ -116,13 +118,13 @@ mysql -u root -p # Convert all tables to use the InnoDB storage engine (added in GitLab 6.8) SELECT CONCAT('ALTER TABLE gitlabhq_production.', table_name, ' ENGINE=InnoDB;') AS 'Copy & run these SQL statements:' FROM information_schema.tables WHERE table_schema = 'gitlabhq_production' AND `ENGINE` <> 'InnoDB' AND `TABLE_TYPE` = 'BASE TABLE'; -# If previous query returned results, copy & run all outputed SQL statements +# If previous query returned results, copy & run all shown SQL statements # Convert all tables to correct character set SET foreign_key_checks = 0; SELECT CONCAT('ALTER TABLE gitlabhq_production.', table_name, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;') AS 'Copy & run these SQL statements:' FROM information_schema.tables WHERE table_schema = 'gitlabhq_production' AND `TABLE_COLLATION` <> 'utf8_unicode_ci' AND `TABLE_TYPE` = 'BASE TABLE'; -# If previous query returned results, copy & run all outputed SQL statements +# If previous query returned results, copy & run all shown SQL statements # turn foreign key checks back on SET foreign_key_checks = 1; @@ -167,6 +169,10 @@ mysql> \q # Set production -> password: the password your replaced $password with earlier sudo -u git -H editor /home/git/gitlab/config/database.yml +# Start GitLab +sudo service gitlab start +sudo service nginx restart + # Run thorough check sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production ``` diff --git a/doc/update/7.4-to-7.5.md b/doc/update/7.4-to-7.5.md new file mode 100644 index 00000000000..673eab3c56e --- /dev/null +++ b/doc/update/7.4-to-7.5.md @@ -0,0 +1,108 @@ +# From 7.4 to 7.5 + +### 0. Stop server + + sudo service gitlab stop + +### 1. Backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 2. Get latest code + +```bash +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +``` + +For GitLab Community Edition: + +```bash +sudo -u git -H git checkout 7-5-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +sudo -u git -H git checkout 7-5-stable-ee +``` + +### 3. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell +sudo -u git -H git fetch +sudo -u git -H git checkout v2.2.0 +``` + +### 4. Install libs, migrations, etc. + +```bash +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without ... postgres') +sudo -u git -H bundle install --without development test postgres --deployment + +# PostgreSQL installations (note: the line below states '--without ... mysql') +sudo -u git -H bundle install --without development test mysql --deployment + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Clean up assets and cache +sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production + +# Update init.d script +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +### 5. Update config files + +#### New configuration options for gitlab.yml + +There are new configuration options available for gitlab.yml. View them with the command below and apply them to your current gitlab.yml. + +``` +git diff origin/7-4-stable:config/gitlab.yml.example origin/7-5-stable:config/gitlab.yml.example +``` + +#### Change Nginx settings + +* HTTP setups: Make `/etc/nginx/sites-available/gitlab` the same as [`lib/support/nginx/gitlab`](/lib/support/nginx/gitlab) but with your settings +* HTTPS setups: Make `/etc/nginx/sites-available/gitlab-ssl` the same as [`lib/support/nginx/gitlab-ssl`](/lib/support/nginx/gitlab-ssl) but with your setting + +### 6. Start application + + sudo service gitlab start + sudo service nginx restart + +### 7. Check application status + +Check if GitLab and its environment are configured correctly: + + sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production + +To make sure you didn't miss anything run a more thorough check with: + + sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production + +If all items are green, then congratulations upgrade is complete! + +## Things went south? Revert to previous version (7.4) + +### 1. Revert the code to the previous version +Follow the [upgrade guide from 7.3 to 7.4](7.3-to-7.4.md), except for the database migration +(The backup is already migrated to the previous version) + +### 2. Restore from the backup: + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` +If you have more than one backup *.tar file(s) please add `BACKUP=timestamp_of_backup` to the command above. diff --git a/doc/update/7.5-to-7.6.md b/doc/update/7.5-to-7.6.md new file mode 100644 index 00000000000..35cd437fdc4 --- /dev/null +++ b/doc/update/7.5-to-7.6.md @@ -0,0 +1,114 @@ +# From 7.5 to 7.6 + +### 0. Stop server + + sudo service gitlab stop + +### 1. Backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 2. Get latest code + +```bash +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +``` + +For GitLab Community Edition: + +```bash +sudo -u git -H git checkout 7-6-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +sudo -u git -H git checkout 7-6-stable-ee +``` + +### 3. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell +sudo -u git -H git fetch +sudo -u git -H git checkout v2.4.0 +``` + +### 4. Install libs, migrations, etc. + +```bash +sudo apt-get install libkrb5-dev + +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without ... postgres') +sudo -u git -H bundle install --without development test postgres --deployment + +# PostgreSQL installations (note: the line below states '--without ... mysql') +sudo -u git -H bundle install --without development test mysql --deployment + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Clean up assets and cache +sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production + +# Update init.d script +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +### 5. Update config files + +#### New configuration options for `gitlab.yml` + +There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). View them with the command below and apply them to your current `gitlab.yml`. + +``` +git diff origin/7-5-stable:config/gitlab.yml.example origin/7-6-stable:config/gitlab.yml.example +``` + +#### Change Nginx settings + +* HTTP setups: Make `/etc/nginx/sites-available/gitlab` the same as [`lib/support/nginx/gitlab`](/lib/support/nginx/gitlab) but with your settings +* HTTPS setups: Make `/etc/nginx/sites-available/gitlab-ssl` the same as [`lib/support/nginx/gitlab-ssl`](/lib/support/nginx/gitlab-ssl) but with your setting + +#### Setup time zone (optional) + +Consider setting the time zone in `gitlab.yml` otherwise GitLab will default to UTC. If you set a time zone previously in [`application.rb`](config/application.rb) (unlikely), unset it. + +### 6. Start application + + sudo service gitlab start + sudo service nginx restart + +### 7. Check application status + +Check if GitLab and its environment are configured correctly: + + sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production + +To make sure you didn't miss anything run a more thorough check with: + + sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production + +If all items are green, then congratulations upgrade is complete! + +## Things went south? Revert to previous version (7.5) + +### 1. Revert the code to the previous version +Follow the [upgrade guide from 7.4 to 7.5](7.4-to-7.5.md), except for the database migration +(The backup is already migrated to the previous version) + +### 2. Restore from the backup: + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` +If you have more than one backup *.tar file(s) please add `BACKUP=timestamp_of_backup` to the command above. diff --git a/doc/update/7.6-to-7.7.md b/doc/update/7.6-to-7.7.md new file mode 100644 index 00000000000..59243713156 --- /dev/null +++ b/doc/update/7.6-to-7.7.md @@ -0,0 +1,119 @@ +# From 7.6 to 7.7 + +### 0. Stop server + + sudo service gitlab stop + +### 1. Backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 2. Get latest code + +```bash +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +``` + +For GitLab Community Edition: + +```bash +sudo -u git -H git checkout 7-7-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +sudo -u git -H git checkout 7-7-stable-ee +``` + +### 3. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell +sudo -u git -H git fetch +sudo -u git -H git checkout v2.4.2 +``` + +### 4. Install libs, migrations, etc. + +```bash +sudo apt-get install libkrb5-dev + +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without ... postgres') +sudo -u git -H bundle install --without development test postgres --deployment + +# PostgreSQL installations (note: the line below states '--without ... mysql') +sudo -u git -H bundle install --without development test mysql --deployment + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Clean up assets and cache +sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production + +# Update init.d script +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +### 5. Update config files + +#### New configuration options for `gitlab.yml` + +There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). View them with the command below and apply them to your current `gitlab.yml`. + +``` +git diff origin/7-6-stable:config/gitlab.yml.example origin/7-7-stable:config/gitlab.yml.example +``` + +#### Change Nginx settings + +* HTTP setups: Make `/etc/nginx/sites-available/gitlab` the same as [`lib/support/nginx/gitlab`](/lib/support/nginx/gitlab) but with your settings +* HTTPS setups: Make `/etc/nginx/sites-available/gitlab-ssl` the same as [`lib/support/nginx/gitlab-ssl`](/lib/support/nginx/gitlab-ssl) but with your setting + +#### Setup time zone (optional) + +Consider setting the time zone in `gitlab.yml` otherwise GitLab will default to UTC. If you set a time zone previously in [`application.rb`](config/application.rb) (unlikely), unset it. + +### 6. Start application + + sudo service gitlab start + sudo service nginx restart + +### 7. Check application status + +Check if GitLab and its environment are configured correctly: + + sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production + +To make sure you didn't miss anything run a more thorough check with: + + sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production + +If all items are green, then congratulations upgrade is complete! + +### 8. GitHub settings (if applicable) + +If you are using GitHub as an OAuth provider for authentication, you should change the callback URL so that it +only contains a root URL (ex. `https://gitlab.example.com/`) + +## Things went south? Revert to previous version (7.6) + +### 1. Revert the code to the previous version +Follow the [upgrade guide from 7.5 to 7.6](7.5-to-7.6.md), except for the database migration +(The backup is already migrated to the previous version) + +### 2. Restore from the backup: + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` +If you have more than one backup *.tar file(s) please add `BACKUP=timestamp_of_backup` to the command above. diff --git a/doc/update/7.7-to-7.8.md b/doc/update/7.7-to-7.8.md new file mode 100644 index 00000000000..46ca163c1bb --- /dev/null +++ b/doc/update/7.7-to-7.8.md @@ -0,0 +1,120 @@ +# From 7.7 to 7.8 + +### 0. Stop server + + sudo service gitlab stop + +### 1. Backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 2. Get latest code + +```bash +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +``` + +For GitLab Community Edition: + +```bash +sudo -u git -H git checkout 7-8-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +sudo -u git -H git checkout 7-8-stable-ee +``` + +### 3. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell +sudo -u git -H git fetch +sudo -u git -H git checkout v2.5.4 +``` + +### 4. Install libs, migrations, etc. + +```bash +sudo apt-get install libkrb5-dev + +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without ... postgres') +sudo -u git -H bundle install --without development test postgres --deployment + +# PostgreSQL installations (note: the line below states '--without ... mysql') +sudo -u git -H bundle install --without development test mysql --deployment + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Clean up assets and cache +sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production + +# Update init.d script +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +### 5. Update config files + +#### New configuration options for `gitlab.yml` + +There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). View them with the command below and apply them to your current `gitlab.yml`. + +``` +git diff origin/7-7-stable:config/gitlab.yml.example origin/7-8-stable:config/gitlab.yml.example +``` + +#### Change Nginx settings + +* HTTP setups: Make `/etc/nginx/sites-available/gitlab` the same as [`lib/support/nginx/gitlab`](/lib/support/nginx/gitlab) but with your settings. +* HTTPS setups: Make `/etc/nginx/sites-available/gitlab-ssl` the same as [`lib/support/nginx/gitlab-ssl`](/lib/support/nginx/gitlab-ssl) but with your settings. +* A new `location /uploads/` section has been added that needs to have the same content as the existing `location @gitlab` section. + +#### Setup time zone (optional) + +Consider setting the time zone in `gitlab.yml` otherwise GitLab will default to UTC. If you set a time zone previously in [`application.rb`](config/application.rb) (unlikely), unset it. + +### 6. Start application + + sudo service gitlab start + sudo service nginx restart + +### 7. Check application status + +Check if GitLab and its environment are configured correctly: + + sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production + +To make sure you didn't miss anything run a more thorough check with: + + sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production + +If all items are green, then congratulations upgrade is complete! + +### 8. GitHub settings (if applicable) + +If you are using GitHub as an OAuth provider for authentication, you should change the callback URL so that it +only contains a root URL (ex. `https://gitlab.example.com/`) + +## Things went south? Revert to previous version (7.7) + +### 1. Revert the code to the previous version +Follow the [upgrade guide from 7.6 to 7.7](7.6-to-7.7.md), except for the database migration +(The backup is already migrated to the previous version) + +### 2. Restore from the backup: + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` +If you have more than one backup *.tar file(s) please add `BACKUP=timestamp_of_backup` to the command above. diff --git a/doc/update/README.md b/doc/update/README.md index 30e9137d7b7..0472537eeb5 100644 --- a/doc/update/README.md +++ b/doc/update/README.md @@ -4,13 +4,13 @@ Depending on the installation method and your GitLab version, there are multiple - [Omnibus update guide](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/update.md) contains the steps needed to update a GitLab [package](https://about.gitlab.com/downloads/). -## Manual Installation +## Installation from source -- [The individual upgrade guides](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update) are for those who have installed GitLab manually. +- [The individual upgrade guides](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update) are for those who have installed GitLab from source. - [The CE to EE update guides](https://gitlab.com/subscribers/gitlab-ee/tree/master/doc/update) are for subscribers of the Enterprise Edition only. The steps are very similar to a version upgrade: stop the server, get the code, update config files for the new functionality, install libs and do migrations, update the init script, start the application and check the application status. -- [Upgrader](upgrader.md) is an automatic ruby script that performs the update for manual installations. +- [Upgrader](upgrader.md) is an automatic ruby script that performs the update for installations from source. - [Patch versions](patch_versions.md) guide includes the steps needed for a patch version, eg. 6.2.0 to 6.2.1. ## Miscellaneous -- [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating your database from MySQL to PostrgreSQL. +- [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating your database from MySQL to PostgreSQL. diff --git a/doc/update/mysql_to_postgresql.md b/doc/update/mysql_to_postgresql.md index 695c083d361..6af940cca36 100644 --- a/doc/update/mysql_to_postgresql.md +++ b/doc/update/mysql_to_postgresql.md @@ -1,4 +1,5 @@ # Migrating GitLab from MySQL to Postgres +*Make sure you view this [guide from the `master` branch](../../../master/doc/update/mysql_to_postgresql.md) for the most up to date instructions.* If you are replacing MySQL with Postgres while keeping GitLab on the same server all you need to do is to export from MySQL, import into Postgres and rebuild the indexes as described below. If you are also moving GitLab to another server, or if you are switching to omnibus-gitlab, you may want to use a GitLab backup file. The second part of this documents explains the procedure to do this. @@ -13,7 +14,7 @@ sudo service gitlab stop git clone https://github.com/gitlabhq/mysql-postgresql-converter.git cd mysql-postgresql-converter -mysqldump --compatible=postgresql --default-character-set=utf8 -r databasename.mysql -u root gitlabhq_production +mysqldump --compatible=postgresql --default-character-set=utf8 -r databasename.mysql -u root gitlabhq_production -p python db_converter.py databasename.mysql databasename.psql # Import the database dump as the application database user @@ -94,7 +95,7 @@ sudo -u git -H mv tmp/backups/TIMESTAMP_gitlab_backup.tar tmp/backups/postgresql # Create a separate database dump with PostgreSQL compatibility cd tmp/backups/postgresql -sudo -u git -H mysqldump --compatible=postgresql --default-character-set=utf8 -r gitlabhq_production.mysql -u root gitlabhq_production +sudo -u git -H mysqldump --compatible=postgresql --default-character-set=utf8 -r gitlabhq_production.mysql -u root gitlabhq_production -p # Clone the database converter sudo -u git -H git clone https://github.com/gitlabhq/mysql-postgresql-converter.git diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index 629c46ad030..ad302492556 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -1,4 +1,5 @@ # Universal update guide for patch versions +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/patch_versions.md) for the most up to date instructions.* For example from 6.2.0 to 6.2.1, also see the [semantic versioning specification](http://semver.org/). diff --git a/doc/update/upgrader.md b/doc/update/upgrader.md index cf59b0e461c..4ed35b2b562 100644 --- a/doc/update/upgrader.md +++ b/doc/update/upgrader.md @@ -1,4 +1,5 @@ # GitLab Upgrader +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/upgrader.md) for the most up to date instructions.* GitLab Upgrader - a ruby script that allows you easily upgrade GitLab to latest minor version. @@ -10,6 +11,8 @@ If you have local changes to your GitLab repository the script will stash them a **GitLab Upgrader is available only for GitLab version 6.4.2 or higher.** +**This script does NOT update gitlab-shell, it needs manual update. See step 5 below.** + ## 0. Backup cd /home/git/gitlab @@ -21,7 +24,7 @@ If you have local changes to your GitLab repository the script will stash them a ## 2. Run GitLab upgrade tool -Note: GitLab 7.2 adds `pkg-config` and `cmake` as dependency. Please check the dependencies in the [installation guide.](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md#1-packages-dependencies) +Note: GitLab 7.6 adds `libkrb5-dev` as a dependency (installed by default on Ubuntu and OSX) while 7.2 adds `pkg-config` and `cmake` as dependency. Please check the dependencies in the [installation guide.](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md#1-packages-dependencies) # Starting with GitLab version 7.0 upgrader script has been moved to bin directory cd /home/git/gitlab @@ -43,28 +46,31 @@ Check if GitLab and its dependencies are configured correctly: If all items are green, then congratulations upgrade is complete! -## 5. Upgrade GitLab Shell (if needed) - -If the `gitlab:check` task reports an outdated version of `gitlab-shell` you should upgrade it. +## 5. Upgrade GitLab Shell -Upgrade it by running the commands below after replacing 2.0.1 with the correct version number: +GitLab Shell might be outdated, running the commands below ensures you're using a compatible version: ``` cd /home/git/gitlab-shell sudo -u git -H git fetch -sudo -u git -H git checkout v2.0.1 +sudo -u git -H git checkout v`cat /home/git/gitlab/GITLAB_SHELL_VERSION` ``` ## One line upgrade command You've read through the entire guide and probably already did all the steps one by one. -Here is a one line command with step 1 to 4 for the next time you upgrade: +Here is a one line command with step 1 to 5 for the next time you upgrade: ```bash -cd /home/git/gitlab; sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production; \ +cd /home/git/gitlab; \ + sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production; \ sudo service gitlab stop; \ if [ -f bin/upgrade.rb ]; then sudo -u git -H ruby bin/upgrade.rb -y; else sudo -u git -H ruby script/upgrade.rb -y; fi; \ + cd /home/git/gitlab-shell; \ + sudo -u git -H git fetch; \ + sudo -u git -H git checkout v`cat /home/git/gitlab/GITLAB_SHELL_VERSION`; \ + cd /home/git/gitlab; \ sudo service gitlab start; \ sudo service nginx restart; sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production ``` diff --git a/doc/web_hooks/web_hooks.md b/doc/web_hooks/web_hooks.md index 31791da8074..29ef5b59bac 100644 --- a/doc/web_hooks/web_hooks.md +++ b/doc/web_hooks/web_hooks.md @@ -24,16 +24,19 @@ Triggered when you push to the repository except when pushing tags. "project_id": 15, "repository": { "name": "Diaspora", - "url": "git@example.com:diaspora.git", + "url": "git@example.com:mike/diasporadiaspora.git", "description": "", - "homepage": "http://example.com/diaspora" + "homepage": "http://example.com/mike/diaspora", + "git_http_url":"http://example.com/mike/diaspora.git", + "git_ssh_url":"git@example.com:mike/diaspora.git", + "visibility_level":0 }, "commits": [ { "id": "b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327", "message": "Update Catalan translation to e38cb41.", "timestamp": "2011-12-12T14:27:31+02:00", - "url": "http://example.com/diaspora/commits/b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327", + "url": "http://example.com/mike/diaspora/commit/b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327", "author": { "name": "Jordi Mallach", "email": "jordi@softcatala.org" @@ -43,7 +46,7 @@ Triggered when you push to the repository except when pushing tags. "id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", "message": "fixed readme", "timestamp": "2012-01-03T23:36:29+02:00", - "url": "http://example.com/diaspora/commits/da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "url": "http://example.com/mike/diaspora/commit/da1560886d4f094c3e6c9ef40349f7d38b5d27d7", "author": { "name": "GitLab dev user", "email": "gitlabdev@dv6700.(none)" @@ -54,6 +57,34 @@ Triggered when you push to the repository except when pushing tags. } ``` +## Tag events + +Triggered when you create (or delete) tags to the repository. + +**Request body:** + +```json +{ + "ref": "refs/tags/v1.0.0", + "before": "0000000000000000000000000000000000000000", + "after": "82b3d5ae55f7080f1e6022629cdb57bfae7cccc7", + "user_id": 1, + "user_name": "John Smith", + "project_id": 1, + "repository": { + "name": "jsmith", + "url": "ssh://git@example.com/jsmith/example.git", + "description": "", + "homepage": "http://example.com/jsmith/example", + "git_http_url":"http://example.com/jsmith/example.git", + "git_ssh_url":"git@example.com:jsmith/example.git", + "visibility_level":0 + }, + "commits": [], + "total_commits_count": 0 +} +``` + ## Issues events Triggered when a new issue is created or an existing issue was updated/closed/reopened. @@ -63,6 +94,11 @@ Triggered when a new issue is created or an existing issue was updated/closed/re ```json { "object_kind": "issue", + "user": { + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" + }, "object_attributes": { "id": 301, "title": "New API: create/update/delete file", @@ -92,6 +128,11 @@ Triggered when a new merge request is created or an existing merge request was u ```json { "object_kind": "merge_request", + "user": { + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" + }, "object_attributes": { "id": 99, "target_branch": "master", @@ -133,7 +174,9 @@ Triggered when a new merge request is created or an existing merge request was u "name": "GitLab dev user", "email": "gitlabdev@dv6700.(none)" } - } + }, + "url": "http://example.com/diaspora/merge_requests/1", + "action": "open" } } ``` diff --git a/doc/workflow/README.md b/doc/workflow/README.md index c26d85e9955..6e70235f5b8 100644 --- a/doc/workflow/README.md +++ b/doc/workflow/README.md @@ -1,4 +1,6 @@ -- [Workflow](workflow.md) +# Workflow + +- [Feature branch workflow](workflow.md) - [Project Features](project_features.md) - [Authorization for merge requests](authorization_for_merge_requests.md) - [Groups](groups.md) @@ -6,3 +8,7 @@ - [GitLab Flow](gitlab_flow.md) - [Notifications](notifications.md) - [Migrating from SVN to GitLab](migrating_from_svn.md) +- [Project importing from GitHub to GitLab](import_projects_from_github.md) +- [Project importing from GitLab.com to your private GitLab instance](import_projects_from_gitlab_com.md) +- [Protected branches](protected_branches.md) +- [Web Editor](web_editor.md) diff --git a/doc/workflow/github_importer/importer.png b/doc/workflow/github_importer/importer.png Binary files differnew file mode 100644 index 00000000000..57636717571 --- /dev/null +++ b/doc/workflow/github_importer/importer.png diff --git a/doc/workflow/github_importer/new_project_page.png b/doc/workflow/github_importer/new_project_page.png Binary files differnew file mode 100644 index 00000000000..002f22d81d7 --- /dev/null +++ b/doc/workflow/github_importer/new_project_page.png diff --git a/doc/workflow/gitlab_flow.md b/doc/workflow/gitlab_flow.md index f8fd7c97e2a..0e87dc74217 100644 --- a/doc/workflow/gitlab_flow.md +++ b/doc/workflow/gitlab_flow.md @@ -1,6 +1,6 @@  -# Introduction +## Introduction Version management with git makes branching and merging much easier than older versioning systems such as SVN. This allows a wide variety of branching strategies and workflows. @@ -29,9 +29,9 @@ People have a hard time figuring out which branch they should develop on or depl Frequently the reaction to this problem is to adopt a standardized pattern such as [git flow](http://nvie.com/posts/a-successful-git-branching-model/) and [GitHub flow](http://scottchacon.com/2011/08/31/github-flow.html) We think there is still room for improvement and will detail a set of practices we call GitLab flow. -# Git flow and its problems +## Git flow and its problems -[ +[ Git flow was one of the first proposals to use git branches and it has gotten a lot of attention. It advocates a master branch and a separate develop branch as well as supporting branches for features, releases and hotfixes. @@ -43,14 +43,14 @@ Since most tools automatically make the master branch the default one and displa The second problem of git flow is the complexity introduced by the hotfix and release branches. These branches can be a good idea for some organizations but are overkill for the vast majority of them. Nowadays most organizations practice continuous delivery which means that your default branch can be deployed. -This means that hotfixed and release branches can be prevented including all the ceremony they introduce. +This means that hotfix and release branches can be prevented including all the ceremony they introduce. An example of this ceremony is the merging back of release branches. Though specialized tools do exist to solve this, they require documentation and add complexity. Frequently developers make a mistake and for example changes are only merged into master and not into the develop branch. The root cause of these errors is that git flow is too complex for most of the use cases. And doing releases doesn't automatically mean also doing hotfixes. -# GitHub flow as a simpler alternative +## GitHub flow as a simpler alternative  @@ -62,13 +62,13 @@ Merging everything into the master branch and deploying often means you minimize But this flow still leaves a lot of questions unanswered regarding deployments, environments, releases and integrations with issues. With GitLab flow we offer additional guidance for these questions. -# Production branch with GitLab flow +## Production branch with GitLab flow  GitHub flow does assume you are able to deploy to production every time you merge a feature branch. This is possible for SaaS applications but are many cases where this is not possible. -One would be a situation where you are not in control of the exact release moment, for example an iOS application that needs to pass AppStore validation. +One would be a situation where you are not in control of the exact release moment, for example an iOS application that needs to pass App Store validation. Another example is when you have deployment windows (workdays from 10am to 4pm when the operations team is at full capacity) but you also merge code at other times. In these cases you can make a production branch that reflects the deployed code. You can deploy a new version by merging in master to the production branch. @@ -78,7 +78,7 @@ This time is pretty accurate if you automatically deploy your production branch. If you need a more exact time you can have your deployment script create a tag on each deployment. This flow prevents the overhead of releasing, tagging and merging that is common to git flow. -# Environment branches with GitLab flow +## Environment branches with GitLab flow  @@ -93,14 +93,14 @@ If master is good to go (it should be if you a practicing [continuous delivery]( If this is not possible because more manual testing is required you can send merge requests from the feature branch to the downstream branches. An 'extreme' version of environment branches are setting up an environment for each feature branch as done by [Teatro](http://teatro.io/). -# Release branches with GitLab flow +## Release branches with GitLab flow - + Only in case you need to release software to the outside world you need to work with release branches. In this case, each branch contains a minor version (2-3-stable, 2-4-stable, etc.). The stable branch uses master as a starting point and is created as late as possible. -By branching as late as possible you minimize the time you have to apply bugfixes to multiple branches. +By branching as late as possible you minimize the time you have to apply bug fixes to multiple branches. After a release branch is announced, only serious bug fixes are included in the release branch. If possible these bug fixes are first merged into master and then cherry-picked into the release branch. This way you can't forget to cherry-pick them into master and encounter the same bug on subsequent releases. @@ -109,7 +109,7 @@ Every time a bug-fix is included in a release branch the patch version is raised Some projects also have a stable branch that points to the same commit as the latest released branch. In this flow it is not common to have a production branch (or git flow master branch). -# Merge/pull requests with GitLab flow +## Merge/pull requests with GitLab flow  @@ -134,7 +134,7 @@ If the assigned person does not feel comfortable they can close the merge reques In GitLab it is common to protect the long-lived branches (e.g. the master branch) so that normal developers [can't modify these protected branches](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/permissions/permissions.md). So if you want to merge it into a protected branch you assign it to someone with master authorizations. -# Issues with GitLab flow +## Issues with GitLab flow  @@ -168,7 +168,7 @@ In this case it is no problem to reuse the same branch name since it was deleted At any time there is at most one branch for every issue. It is possible that one feature branch solves more than one issue. -# Linking and closing issues from merge requests +## Linking and closing issues from merge requests  @@ -177,11 +177,11 @@ In GitLab this creates a comment in the issue that the merge requests mentions t And the merge request shows the linked issues. These issues are closed once code is merged into the default branch. -If you only want to make the reference without closing the issue you can also just mention it: "Ducktyping is preferred. #12". +If you only want to make the reference without closing the issue you can also just mention it: "Duck typing is preferred. #12". If you have an issue that spans across multiple repositories, the best thing is to create an issue for each repository and link all issues to a parent issue. -# Squashing commits with rebase +## Squashing commits with rebase  @@ -189,7 +189,7 @@ With git you can use an interactive rebase (rebase -i) to squash multiple commit This functionality is useful if you made a couple of commits for small changes during development and want to replace them with a single commit or if you want to make the order more logical. However you should never rebase commits you have pushed to a remote server. Somebody can have referred to the commits or cherry-picked them. -When you rebase you change the identifier (SHA1) of the commit and this is confusing. +When you rebase you change the identifier (SHA-1) of the commit and this is confusing. If you do that the same change will be known under multiple identifiers and this can cause much confusion. If people already reviewed your code it will be hard for them to review only the improvements you made since then if you have rebased everything into one commit. @@ -207,7 +207,7 @@ If you revert a merge and you change your mind, revert the revert instead of mer Being able to revert a merge is a good reason always to create a merge commit when you merge manually with the `--no-ff` option. Git management software will always create a merge commit when you accept a merge request. -# Do not order commits with rebase +## Do not order commits with rebase  @@ -231,8 +231,8 @@ The last reason for creating merge commits is having long lived branches that yo Martin Fowler, in [his article about feature branches](http://martinfowler.com/bliki/FeatureBranch.html) talks about this Continuous Integration (CI). At GitLab we are guilty of confusing CI with branch testing. Quoting Martin Fowler: "I've heard people say they are doing CI because they are running builds, perhaps using a CI server, on every branch with every commit. That's continuous building, and a Good Thing, but there's no integration, so it's not CI.". -The solution to prevent many merge commits is to keep your feature branches shortlived, the vast majority should take less than one day of work. -If your feature branches commenly take more than a day of work, look into ways to create smaller units of work and/or use [feature toggles](http://martinfowler.com/bliki/FeatureToggle.html). +The solution to prevent many merge commits is to keep your feature branches short-lived, the vast majority should take less than one day of work. +If your feature branches commonly take more than a day of work, look into ways to create smaller units of work and/or use [feature toggles](http://martinfowler.com/bliki/FeatureToggle.html). As for the long running branches that take more than one day there are two strategies. In a CI strategy you can merge in master at the start of the day to prevent painful merges at a later time. In a synchronization point strategy you only merge in from well defined points in time, for example a tagged release. @@ -244,7 +244,7 @@ Developing software happen in small messy steps and it is OK to have your histor You can use tools to view the network graphs of commits and understand the messy history that created your code. If you rebase code the history is incorrect, and there is no way for tools to remedy this because they can't deal with changing commit identifiers. -# Voting on merge requests +## Voting on merge requests  @@ -252,7 +252,7 @@ It is common to voice approval or disapproval by using +1 or -1 emoticons. In GitLab the +1 and -1 are aggregated and shown at the top of the merge request. As a rule of thumb anything that doesn't have two times more +1's than -1's is suspect and should not be merged yet. -# Pushing and removing branches +## Pushing and removing branches  @@ -266,7 +266,7 @@ This ensures that the branch overview in the repository management software show This also ensures that when someone reopens the issue a new branch with the same name can be used without problem. When you reopen an issue you need to create a new merge request. -# Committing often and with the right message +## Committing often and with the right message  @@ -282,7 +282,7 @@ Some words that are bad commit messages because they don't contain munch informa The word fix or fixes is also a red flag, unless it comes after the commit sentence and references an issue number. To see more information about the formatting of commit messages please see this great [blog post by Tim Pope](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html). -# Testing before merging +## Testing before merging  @@ -299,7 +299,7 @@ If there are no merge conflicts and the feature branches are short lived the ris If there are merge conflicts you merge the master branch into the feature branch and the CI server will rerun the tests. If you have long lived feature branches that last for more than a few days you should make your issues smaller. -# Merging in other code +## Merging in other code  diff --git a/doc/workflow/gitlab_importer/importer.png b/doc/workflow/gitlab_importer/importer.png Binary files differnew file mode 100644 index 00000000000..d2a286d8cac --- /dev/null +++ b/doc/workflow/gitlab_importer/importer.png diff --git a/doc/workflow/gitlab_importer/new_project_page.png b/doc/workflow/gitlab_importer/new_project_page.png Binary files differnew file mode 100644 index 00000000000..5e239208e1e --- /dev/null +++ b/doc/workflow/gitlab_importer/new_project_page.png diff --git a/doc/workflow/import_projects_from_github.md b/doc/workflow/import_projects_from_github.md new file mode 100644 index 00000000000..8644b4ffc73 --- /dev/null +++ b/doc/workflow/import_projects_from_github.md @@ -0,0 +1,13 @@ +# Project importing from GitHub to GitLab + +You can import your existing GitHub projects to GitLab. But keep in mind that it is possible only if +GitHub support is enabled on your GitLab instance. You can read more about GitHub support [here](http://doc.gitlab.com/ce/integration/github.html) +To get to the importer page you need to go to "New project" page. + + + +Click on the "Import project from GitHub" link and you will be redirected to GitHub for permission to access your projects. After accepting, you'll be automatically redirected to the importer. + + + +To import a project, you can simple click "Add". The importer will import your repository and issues. Once the importer is done, a new GitLab project will be created with your imported data.
\ No newline at end of file diff --git a/doc/workflow/import_projects_from_gitlab_com.md b/doc/workflow/import_projects_from_gitlab_com.md new file mode 100644 index 00000000000..f4c4e955d46 --- /dev/null +++ b/doc/workflow/import_projects_from_gitlab_com.md @@ -0,0 +1,18 @@ +# Project importing from GitLab.com to your private GitLab instance + +You can import your existing GitLab.com projects to your GitLab instance. But keep in mind that it is possible only if +GitLab support is enabled on your GitLab instance. +You can read more about Gitlab support [here](http://doc.gitlab.com/ce/integration/gitlab.html) +To get to the importer page you need to go to "New project" page. + + + +Click on the "Import projects from Gitlab.com" link and you will be redirected to GitLab.com +for permission to access your projects. After accepting, you'll be automatically redirected to the importer. + + + + + +To import a project, you can simple click "Import". The importer will import your repository and issues. +Once the importer is done, a new GitLab project will be created with your imported data.
\ No newline at end of file diff --git a/doc/workflow/migrating_from_svn.md b/doc/workflow/migrating_from_svn.md index 207e3641802..485db4834e9 100644 --- a/doc/workflow/migrating_from_svn.md +++ b/doc/workflow/migrating_from_svn.md @@ -3,7 +3,7 @@ SVN stands for Subversion and is a version control system (VCS). Git is a distributed version control system. -There are some major differences between the two, for more information consult your favourite search engine. +There are some major differences between the two, for more information consult your favorite search engine. Git has tools for migrating SVN repositories to git, namely `git svn`. You can read more about this at [git documentation pages](http://git-scm.com/book/en/Git-and-Other-Systems-Git-and-Subversion). diff --git a/doc/workflow/notifications.md b/doc/workflow/notifications.md index 3c3ce162df5..17215de677e 100644 --- a/doc/workflow/notifications.md +++ b/doc/workflow/notifications.md @@ -24,14 +24,14 @@ Each of these settings have levels of notification: #### Global Settings Global Settings are at the bottom of the hierarchy. -Any setting set here will be overriden by a setting at the group or a project level. +Any setting set here will be overridden by a setting at the group or a project level. Group or Project settings can use `global` notification setting which will then use anything that is set at Global Settings. #### Group Settings -Group Settings are taking presedence over Global Settings but are on a level below Project Settings. +Group Settings are taking precedence over Global Settings but are on a level below Project Settings. This means that you can set a different level of notifications per group while still being able to have a finer level setting per project. Organization like this is suitable for users that belong to different groups but don't have the @@ -39,7 +39,7 @@ same need for being notified for every group they are member of. #### Project Settings -Project Settings are at the top level and any setting placed at this level will take presedence of any +Project Settings are at the top level and any setting placed at this level will take precedence of any other setting. This is suitable for users that have different needs for notifications per project basis. diff --git a/doc/workflow/protected_branches.md b/doc/workflow/protected_branches.md new file mode 100644 index 00000000000..805f7f8d35c --- /dev/null +++ b/doc/workflow/protected_branches.md @@ -0,0 +1,33 @@ +# Protected branches + +Permission in GitLab are fundamentally defined around the idea of having read or write permission to the repository and branches. + +To prevent people from messing with history or pushing code without review, we've created protected branches. + +A protected branch does three simple things: + +* it prevents pushes from everybody except users with Master permission +* it prevents anyone from force pushing to the branch +* it prevents anyone from deleting the branch + +You can make any branch a protected branch. GitLab makes the master branch a protected branch by default. + +To protect a branch, user needs to have at least a Master permission level, see [permissions document](permissions/permissions.md). + + + +Navigate to project settings page and select `protected branches`. From the `Branch` dropdown menu select the branch you want to protect. + +Some workflows, like [GitLab workflow](gitlab_flow.md), require all users with write access to submit a Merge request in order to get the code into a protected branch. + +Since Masters and Owners can already push to protected branches, that means Developers cannot push to protected branch and need to submit a Merge request. + +However, there are workflows where that is not needed and only protecting from force pushes and branch removal is useful. + +For those workflows, you can allow everyone with write access to push to a protected branch by selecting `Developers can push` check box. + +On already protected branches you can also allow developers to push to the repository by selecting the `Developers can push` check box. + + + + diff --git a/doc/workflow/protected_branches/protected_branches1.png b/doc/workflow/protected_branches/protected_branches1.png Binary files differnew file mode 100644 index 00000000000..5c2a3de5f70 --- /dev/null +++ b/doc/workflow/protected_branches/protected_branches1.png diff --git a/doc/workflow/protected_branches/protected_branches2.png b/doc/workflow/protected_branches/protected_branches2.png Binary files differnew file mode 100644 index 00000000000..2dca3541365 --- /dev/null +++ b/doc/workflow/protected_branches/protected_branches2.png diff --git a/doc/workflow/web_editor.md b/doc/workflow/web_editor.md new file mode 100644 index 00000000000..7fc8f96b9ec --- /dev/null +++ b/doc/workflow/web_editor.md @@ -0,0 +1,26 @@ +# GitLab Web Editor + +In GitLab you can create new files and edit existing files using our web editor. +This is especially useful if you don't have access to a command line or you just want to do a quick fix. +You can easily access the web editor, depending on the context. +Let's start from newly created project. + +Click on `Add a file` +to create the first file and open it in the web editor. + + + +Fill in a file name, some content, a commit message, branch name and press the commit button. +The file will be saved to the repository. + + + +You can edit any text file in a repository by pressing the edit button, when +viewing the file. + + + +Editing a file is almost the same as creating a new file, +with as addition the ability to preview your changes in a separate tab. Also you can save your change to another branch by filling out field `branch` + + diff --git a/doc/workflow/web_editor/edit_file.png b/doc/workflow/web_editor/edit_file.png Binary files differnew file mode 100644 index 00000000000..f480c69ac3e --- /dev/null +++ b/doc/workflow/web_editor/edit_file.png diff --git a/doc/workflow/web_editor/empty_project.png b/doc/workflow/web_editor/empty_project.png Binary files differnew file mode 100644 index 00000000000..6a049f6beaf --- /dev/null +++ b/doc/workflow/web_editor/empty_project.png diff --git a/doc/workflow/web_editor/new_file.png b/doc/workflow/web_editor/new_file.png Binary files differnew file mode 100644 index 00000000000..55ebd9e0257 --- /dev/null +++ b/doc/workflow/web_editor/new_file.png diff --git a/doc/workflow/web_editor/show_file.png b/doc/workflow/web_editor/show_file.png Binary files differnew file mode 100644 index 00000000000..9cafcb55109 --- /dev/null +++ b/doc/workflow/web_editor/show_file.png diff --git a/doc/workflow/workflow.md b/doc/workflow/workflow.md index ab29cfb670b..f70e41df842 100644 --- a/doc/workflow/workflow.md +++ b/doc/workflow/workflow.md @@ -1,4 +1,4 @@ -# Workflow +# Feature branch workflow 1. Clone project: |
