diff options
Diffstat (limited to 'doc')
138 files changed, 3829 insertions, 772 deletions
diff --git a/doc/README.md b/doc/README.md index 2f90cf14a64..4e00dceac2b 100644 --- a/doc/README.md +++ b/doc/README.md @@ -2,24 +2,30 @@ ## User documentation -- [API](api/README.md) Explore how you can access GitLab via a simple and powerful API. -- [Markdown](markdown/markdown.md) Learn what you can do with GitLab's advanced formatting system. +- [API](api/README.md) Automate GitLab via a simple and powerful API. +- [Markdown](markdown/markdown.md) GitLab's advanced formatting system. - [Permissions](permissions/permissions.md) Learn what each role in a project (guest/reporter/developer/master/owner) can do. -- [Public access](public_access/public_access.md) Learn how you can allow public and internal access to a project. +- [Project Services](project_services/project_services.md) Integrate a project with external services, such as CI and chat. +- [Public access](public_access/public_access.md) Learn how you can allow public and internal access to projects. - [SSH](ssh/README.md) Setup your ssh keys and deploy keys for secure access to your projects. - [Web hooks](web_hooks/web_hooks.md) Let GitLab notify you when new code has been pushed to your project. -- [Workflow](workflow/README.md) Learn how to use Git and GitLab together. +- [Workflow](workflow/README.md) Using GitLab functionality and importing projects from GitHub and SVN. +- [GitLab as OAuth2 authentication service provider](integration/oauth_provider.md). It allows you to login to other applications from GitLab. ## Administrator documentation -- [Install](install/README.md) Requirements, directory structures and manual installation. +- [Install](install/README.md) Requirements, directory structures and installation from source. - [Integration](integration/README.md) How to integrate with systems such as JIRA, Redmine, LDAP and Twitter. -- [Raketasks](raketasks/README.md) Explore what GitLab has in store for you to make administration easier. -- [System hooks](system_hooks/system_hooks.md) Let GitLab notify you when certain management tasks need to be carried out. +- [Raketasks](raketasks/README.md) Backups, maintenance, automatic web hook setup and the importing of projects. +- [Custom git hooks](hooks/custom_hooks.md) Custom git hooks (on the filesystem) for when web hooks aren't enough. +- [System hooks](system_hooks/system_hooks.md) Notifications when users, projects and keys are changed. - [Security](security/README.md) Learn what you can do to further secure your GitLab instance. - [Update](update/README.md) Update guides to upgrade your installation. - [Welcome message](customization/welcome_message.md) Add a custom welcome message to the sign-in page. - [Issue closing](customization/issue_closing.md) Customize how to close an issue from commit messages. +- [Libravatar](customization/libravatar.md) Use Libravatar for user avatars. +- [Operations](operations/README.md) Keeping GitLab up and running +- [Log system](logs/logs.md) Log system ## Contributor documentation diff --git a/doc/api/README.md b/doc/api/README.md index f76a253083f..dec530d0b81 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -21,13 +21,8 @@ ## Clients -- [php-gitlab-api](https://github.com/m4tthumphrey/php-gitlab-api) - PHP -- [Laravel API Wrapper for GitLab CE](https://github.com/adamgoose/gitlab) - PHP / [Laravel](http://laravel.com) -- [Ruby Wrapper](https://github.com/NARKOZ/gitlab) - Ruby -- [python-gitlab](https://github.com/Itxaka/python-gitlab) - Python -- [java-gitlab-api](https://github.com/timols/java-gitlab-api) - Java -- [node-gitlab](https://github.com/moul/node-gitlab) - Node.js -- [NGitLab](https://github.com/Scooletz/NGitLab) - .NET +Find API Clients for GitLab [on our website](https://about.gitlab.com/applications/#api-clients). +You can use [GitLab as an OAuth2 client](oauth2.md) to make API calls. ## Introduction @@ -57,6 +52,24 @@ curl --header "PRIVATE-TOKEN: QVy1PB7sTxfy4pqfZM1U" "http://example.com/api/v3/p The API uses JSON to serialize data. You don't need to specify `.json` at the end of API URL. +## Authentication with OAuth2 token + +Instead of the private_token you can transmit the OAuth2 access token as a header or as a parameter. + +### OAuth2 token (as a parameter) + +``` +curl https://localhost:3000/api/v3/user?access_token=OAUTH-TOKEN +``` + +### OAuth2 token (as a header) + +``` +curl -H "Authorization: Bearer OAUTH-TOKEN" https://localhost:3000/api/v3/user +``` + +Read more about [GitLab as an OAuth2 client](oauth2.md). + ## Status codes The API is designed to return different status codes according to context and action. In this way if a request results in an error the caller is able to get insight into what went wrong, e.g. status code `400 Bad Request` is returned if a required attribute is missing from the request. The following list gives an overview of how the API functions generally behave. @@ -85,7 +98,7 @@ Return values: ## Sudo -All API requests support performing an api call as if you were another user, if your private token is for an administration account. You need to pass `sudo` parameter by url or header with an id or username of the user you want to perform the operation as. If passed as header, the header name must be "SUDO" (capitals). +All API requests support performing an api call as if you were another user, if your private token is for an administration account. You need to pass `sudo` parameter by URL or header with an id or username of the user you want to perform the operation as. If passed as header, the header name must be "SUDO" (capitals). If a non administrative `private_token` is provided then an error message will be returned with status code 403: @@ -130,7 +143,7 @@ When listing resources you can pass the following parameters: - `page` (default: `1`) - page number - `per_page` (default: `20`, max: `100`) - number of items to list per page -[Link headers](http://www.w3.org/wiki/LinkHeader) are send back with each response. These have `rel` prev/next/first/last and contain the relevant URL. Please use these instead of generating your own urls. +[Link headers](http://www.w3.org/wiki/LinkHeader) are send back with each response. These have `rel` prev/next/first/last and contain the relevant URL. Please use these instead of generating your own URLs. ## id vs iid @@ -158,7 +171,7 @@ When an attribute is missing, you will get something like: HTTP/1.1 400 Bad Request Content-Type: application/json - + { "message":"400 (Bad request) \"title\" not given" } @@ -167,7 +180,7 @@ When a validation error occurs, error messages will be different. They will hold HTTP/1.1 400 Bad Request Content-Type: application/json - + { "message": { "bio": [ diff --git a/doc/api/branches.md b/doc/api/branches.md index 74386615545..6a9c10c8520 100644 --- a/doc/api/branches.md +++ b/doc/api/branches.md @@ -15,27 +15,20 @@ Parameters: ```json [ { - "name": "master", "commit": { + "author_email": "john@example.com", + "author_name": "John Smith", + "authored_date": "2012-06-27T05:51:39-07:00", + "committed_date": "2012-06-28T03:44:20-07:00", + "committer_email": "john@example.com", + "committer_name": "John Smith", "id": "7b5c3cc8be40ee161ae89a06bba6229da1032a0c", - "parents": [ - { - "id": "4ad91d3c1144c406e50c7b33bae684bd6837faf8" - } - ], - "tree": "46e82de44b1061621357f24c05515327f2795a95", "message": "add projects API", - "author": { - "name": "John Smith", - "email": "john@example.com" - }, - "committer": { - "name": "John Smith", - "email": "john@example.com" - }, - "authored_date": "2012-06-27T05:51:39-07:00", - "committed_date": "2012-06-28T03:44:20-07:00" + "parent_ids": [ + "4ad91d3c1144c406e50c7b33bae684bd6837faf8" + ] }, + "name": "master", "protected": true } ] @@ -56,27 +49,20 @@ Parameters: ```json { - "name": "master", "commit": { + "author_email": "john@example.com", + "author_name": "John Smith", + "authored_date": "2012-06-27T05:51:39-07:00", + "committed_date": "2012-06-28T03:44:20-07:00", + "committer_email": "john@example.com", + "committer_name": "John Smith", "id": "7b5c3cc8be40ee161ae89a06bba6229da1032a0c", - "parents": [ - { - "id": "4ad91d3c1144c406e50c7b33bae684bd6837faf8" - } - ], - "tree": "46e82de44b1061621357f24c05515327f2795a95", "message": "add projects API", - "author": { - "name": "John Smith", - "email": "john@example.com" - }, - "committer": { - "name": "John Smith", - "email": "john@example.com" - }, - "authored_date": "2012-06-27T05:51:39-07:00", - "committed_date": "2012-06-28T03:44:20-07:00" + "parent_ids": [ + "4ad91d3c1144c406e50c7b33bae684bd6837faf8" + ] }, + "name": "master", "protected": true } ``` @@ -97,27 +83,20 @@ Parameters: ```json { - "name": "master", "commit": { + "author_email": "john@example.com", + "author_name": "John Smith", + "authored_date": "2012-06-27T05:51:39-07:00", + "committed_date": "2012-06-28T03:44:20-07:00", + "committer_email": "john@example.com", + "committer_name": "John Smith", "id": "7b5c3cc8be40ee161ae89a06bba6229da1032a0c", - "parents": [ - { - "id": "4ad91d3c1144c406e50c7b33bae684bd6837faf8" - } - ], - "tree": "46e82de44b1061621357f24c05515327f2795a95", "message": "add projects API", - "author": { - "name": "John Smith", - "email": "john@example.com" - }, - "committer": { - "name": "John Smith", - "email": "john@example.com" - }, - "authored_date": "2012-06-27T05:51:39-07:00", - "committed_date": "2012-06-28T03:44:20-07:00" + "parent_ids": [ + "4ad91d3c1144c406e50c7b33bae684bd6837faf8" + ] }, + "name": "master", "protected": true } ``` @@ -138,27 +117,20 @@ Parameters: ```json { - "name": "master", "commit": { + "author_email": "john@example.com", + "author_name": "John Smith", + "authored_date": "2012-06-27T05:51:39-07:00", + "committed_date": "2012-06-28T03:44:20-07:00", + "committer_email": "john@example.com", + "committer_name": "John Smith", "id": "7b5c3cc8be40ee161ae89a06bba6229da1032a0c", - "parents": [ - { - "id": "4ad91d3c1144c406e50c7b33bae684bd6837faf8" - } - ], - "tree": "46e82de44b1061621357f24c05515327f2795a95", "message": "add projects API", - "author": { - "name": "John Smith", - "email": "john@example.com" - }, - "committer": { - "name": "John Smith", - "email": "john@example.com" - }, - "authored_date": "2012-06-27T05:51:39-07:00", - "committed_date": "2012-06-28T03:44:20-07:00" + "parent_ids": [ + "4ad91d3c1144c406e50c7b33bae684bd6837faf8" + ] }, + "name": "master", "protected": false } ``` @@ -177,21 +149,20 @@ Parameters: ```json { - "name": "my-new-branch", "commit": { - "id": "8848c0e90327a0b70f1865b843fb2fbfb9345e57", - "message": "Merge pull request #54 from brightbox/use_fog_brightbox_module\n\nUpdate to use fog-brightbox module", - "parent_ids": [ - "fff449e0bf453576f16c91d6544f00a2664009d8", - "f93a93626fec20fd659f4ed3ab2e64019b6169ae" - ], - "authored_date": "2014-02-20T19:54:55+02:00", - "author_name": "john smith", "author_email": "john@example.com", - "committed_date": "2014-02-20T19:54:55+02:00", - "committer_name": "john smith", - "committer_email": "john@example.com" + "author_name": "John Smith", + "authored_date": "2012-06-27T05:51:39-07:00", + "committed_date": "2012-06-28T03:44:20-07:00", + "committer_email": "john@example.com", + "committer_name": "John Smith", + "id": "7b5c3cc8be40ee161ae89a06bba6229da1032a0c", + "message": "add projects API", + "parent_ids": [ + "4ad91d3c1144c406e50c7b33bae684bd6837faf8" + ] }, + "name": "master", "protected": false } ``` @@ -211,3 +182,11 @@ Parameters: It return 200 if succeed, 404 if the branch to be deleted does not exist or 400 for other reasons. In case of an error, an explaining message is provided. + +Success response: + +```json +{ + "branch_name": "my-removed-branch" +} +``` diff --git a/doc/api/commits.md b/doc/api/commits.md index 9475ecbaa67..eb8d6a43592 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -93,3 +93,66 @@ Parameters: } ] ``` + +## Get the comments of a commit + +Get the comments of a commit in a project. + +``` +GET /projects/:id/repository/commits/:sha/comments +``` + +Parameters: + +- `id` (required) - The ID of a project +- `sha` (required) - The name of a repository branch or tag or if not given the default branch + +```json +[ + { + "note": "this code is really nice", + "author": { + "id": 11, + "username": "admin", + "email": "admin@local.host", + "name": "Administrator", + "state": "active", + "created_at": "2014-03-06T08:17:35.000Z" + } + } +] +``` + +## Post comment to commit + +Adds a comment to a commit. Optionally you can post comments on a specific line of a commit. Therefor both `path`, `line_new` and `line_old` are required. + +``` +POST /projects/:id/repository/commits/:sha/comments +``` + +Parameters: + +- `id` (required) - The ID of a project +- `sha` (required) - The name of a repository branch or tag or if not given the default branch +- `note` (required) - Text of comment +- `path` (optional) - The file path +- `line` (optional) - The line number +- `line_type` (optional) - The line type (new or old) + +```json +{ + "author": { + "id": 1, + "username": "admin", + "email": "admin@local.host", + "name": "Administrator", + "blocked": false, + "created_at": "2012-04-29T08:46:00Z" + }, + "note": "text1", + "path": "example.rb", + "line": 5, + "line_type": "new" +} +``` diff --git a/doc/api/groups.md b/doc/api/groups.md index 6b379b02d28..b5a4b05ccaf 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -14,11 +14,13 @@ GET /groups "id": 1, "name": "Foobar Group", "path": "foo-bar", - "owner_id": 18 + "description": "An interesting group" } ] ``` +You can search for groups by name or path, see below. + ## Details of a group Get all details of a group. @@ -29,7 +31,7 @@ GET /groups/:id Parameters: -- `id` (required) - The ID of a group +- `id` (required) - The ID or path of a group ## New group @@ -43,6 +45,7 @@ Parameters: - `name` (required) - The name of the group - `path` (required) - The path of the group +- `description` (optional) - The group's description ## Transfer project to group @@ -54,7 +57,7 @@ POST /groups/:id/projects/:project_id Parameters: -- `id` (required) - The ID of a group +- `id` (required) - The ID or path of a group - `project_id` (required) - The ID of a project ## Remove group @@ -67,7 +70,26 @@ DELETE /groups/:id Parameters: -- `id` (required) - The ID of a user group +- `id` (required) - The ID or path of a user group + +## Search for group + +Get all groups that match your string in their name or path. + +``` +GET /groups?search=foobar +``` + +```json +[ + { + "id": 1, + "name": "Foobar Group", + "path": "foo-bar", + "description": "An interesting group" + } +] +``` ## Group members @@ -124,10 +146,24 @@ POST /groups/:id/members Parameters: -- `id` (required) - The ID of a group +- `id` (required) - The ID or path of a group - `user_id` (required) - The ID of a user to add - `access_level` (required) - Project access level +### Edit group team member + +Updates a group team member to a specified access level. + +``` +PUT /groups/:id/members/:user_id +``` + +Parameters: + +- `id` (required) - The ID of a group +- `user_id` (required) - The ID of a group member +- `access_level` (required) - Project access level + ### Remove user team member Removes user from user team. @@ -138,5 +174,5 @@ DELETE /groups/:id/members/:user_id Parameters: -- `id` (required) - The ID of a user group +- `id` (required) - The ID or path of a user group - `user_id` (required) - The ID of a group member diff --git a/doc/api/issues.md b/doc/api/issues.md index ceeb683a6bf..a7dd8b74c35 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -18,6 +18,8 @@ Parameters: - `state` (optional) - Return `all` issues or just those that are `opened` or `closed` - `labels` (optional) - Comma-separated list of label names +- `order_by` (optional) - Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` +- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` ```json [ @@ -56,7 +58,7 @@ Parameters: "title": "v1.0", "description": "", "due_date": "2012-07-20", - "state": "reopenend", + "state": "reopened", "updated_at": "2012-07-04T13:42:48Z", "created_at": "2012-07-04T13:42:48Z" }, @@ -105,6 +107,8 @@ Parameters: - `state` (optional) - Return `all` issues or just those that are `opened` or `closed` - `labels` (optional) - Comma-separated list of label names - `milestone` (optional) - Milestone title +- `order_by` (optional) - Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` +- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` ## Single issue @@ -204,7 +208,7 @@ If an error occurs, an error number and a message explaining the reason is retur ## Delete existing issue (**Deprecated**) -The function is deprecated and returns a `405 Method Not Allowed` error if called. An issue gets now closed and is done by calling `PUT /projects/:id/issues/:issue_id` with parameter `closed` set to 1. +The function is deprecated and returns a `405 Method Not Allowed` error if called. An issue gets now closed and is done by calling `PUT /projects/:id/issues/:issue_id` with parameter `state_event` set to `close`. ``` DELETE /projects/:id/issues/:issue_id diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 14884e53915..1f3fd26a241 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -2,7 +2,9 @@ ## List merge requests -Get all merge requests for this project. The `state` parameter can be used to get only merge requests with a given state (`opened`, `closed`, or `merged`) or all of them (`all`). The pagination parameters `page` and `per_page` can be used to restrict the list of merge requests. +Get all merge requests for this project. +The `state` parameter can be used to get only merge requests with a given state (`opened`, `closed`, or `merged`) or all of them (`all`). +The pagination parameters `page` and `per_page` can be used to restrict the list of merge requests. ``` GET /projects/:id/merge_requests @@ -14,8 +16,8 @@ Parameters: - `id` (required) - The ID of a project - `state` (optional) - Return `all` requests or just those that are `merged`, `opened` or `closed` -- `order_by` (optional) - Return requests ordered by `created_at` or `updated_at` fields -- `sort` (optional) - Return requests sorted in `asc` or `desc` order +- `order_by` (optional) - Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` +- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` ```json [ @@ -94,6 +96,76 @@ Parameters: } ``` +## Get single MR changes + +Shows information about the merge request including its files and changes + +``` +GET /projects/:id/merge_request/:merge_request_id/changes +``` + +Parameters: + +- `id` (required) - The ID of a project +- `merge_request_id` (required) - The ID of MR + +```json +{ + "id": 21, + "iid": 1, + "project_id": 4, + "title": "Blanditiis beatae suscipit hic assumenda et molestias nisi asperiores repellat et.", + "description": "Qui voluptatibus placeat ipsa alias quasi. Deleniti rem ut sint. Optio velit qui distinctio.", + "state": "reopened", + "created_at": "2015-02-02T19:49:39.159Z", + "updated_at": "2015-02-02T20:08:49.959Z", + "target_branch": "secret_token", + "source_branch": "version-1-9", + "upvotes": 0, + "downvotes": 0, + "author": { + "name": "Chad Hamill", + "username": "jarrett", + "id": 5, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/b95567800f828948baf5f4160ebb2473?s=40&d=identicon" + }, + "assignee": { + "name": "Administrator", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40&d=identicon" + }, + "source_project_id": 4, + "target_project_id": 4, + "labels": [ ], + "milestone": { + "id": 5, + "iid": 1, + "project_id": 4, + "title": "v2.0", + "description": "Assumenda aut placeat expedita exercitationem labore sunt enim earum.", + "state": "closed", + "created_at": "2015-02-02T19:49:26.013Z", + "updated_at": "2015-02-02T19:49:26.013Z", + "due_date": null + }, + "files": [ + { + "old_path": "VERSION", + "new_path": "VERSION", + "a_mode": "100644", + "b_mode": "100644", + "diff": "--- a/VERSION\ +++ b/VERSION\ @@ -1 +1 @@\ -1.9.7\ +1.9.8", + "new_file": false, + "renamed_file": false, + "deleted_file": false + } + ] +} +``` + ## Create MR Creates a new merge request. @@ -109,6 +181,7 @@ Parameters: - `target_branch` (required) - The target branch - `assignee_id` (optional) - Assignee user ID - `title` (required) - Title of MR +- `description` (optional) - Description of MR - `target_project_id` (optional) - The target project (numeric id) ```json @@ -160,6 +233,7 @@ Parameters: - `target_branch` - The target branch - `assignee_id` - Assignee user ID - `title` - Title of MR +- `description` - Description of MR - `state_event` - New state (close|reopen|merge) ```json @@ -169,6 +243,7 @@ Parameters: "source_branch": "test1", "project_id": 3, "title": "test1", + "description": "description1", "state": "opened", "upvotes": 0, "downvotes": 0, diff --git a/doc/api/milestones.md b/doc/api/milestones.md index 2f525327504..d48b3bcce8a 100644 --- a/doc/api/milestones.md +++ b/doc/api/milestones.md @@ -72,3 +72,16 @@ Parameters: - `description` (optional) - The description of a milestone - `due_date` (optional) - The due date of the milestone - `state_event` (optional) - The state event of the milestone (close|activate) + +## Get all issues assigned to a single milestone + +Gets all issues assigned to a single project milestone. + +``` +GET /projects/:id/milestones/:milestone_id/issues +``` + +Parameters: + +- `id` (required) - The ID of a project +- `milestone_id` (required) - The ID of a project milestone diff --git a/doc/api/notes.md b/doc/api/notes.md index b5256ac803e..c22e493562a 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -78,6 +78,21 @@ Parameters: - `issue_id` (required) - The ID of an issue - `body` (required) - The content of a note +### Modify existing issue note + +Modify existing note of an issue. + +``` +PUT /projects/:id/issues/:issue_id/notes/:note_id +``` + +Parameters: + +- `id` (required) - The ID of a project +- `issue_id` (required) - The ID of an issue +- `note_id` (required) - The ID of a note +- `body` (required) - The content of a note + ## Snippets ### List all snippet notes @@ -137,7 +152,22 @@ POST /projects/:id/snippets/:snippet_id/notes Parameters: - `id` (required) - The ID of a project -- `snippet_id` (required) - The ID of an snippet +- `snippet_id` (required) - The ID of a snippet +- `body` (required) - The content of a note + +### Modify existing snippet note + +Modify existing note of a snippet. + +``` +PUT /projects/:id/snippets/:snippet_id/notes/:note_id +``` + +Parameters: + +- `id` (required) - The ID of a project +- `snippet_id` (required) - The ID of a snippet +- `note_id` (required) - The ID of a note - `body` (required) - The content of a note ## Merge Requests @@ -199,3 +229,18 @@ Parameters: - `id` (required) - The ID of a project - `merge_request_id` (required) - The ID of a merge request - `body` (required) - The content of a note + +### Modify existing merge request note + +Modify existing note of a merge request. + +``` +PUT /projects/:id/merge_requests/:merge_request_id/notes/:note_id +``` + +Parameters: + +- `id` (required) - The ID of a project +- `merge_request_id` (required) - The ID of a merge request +- `note_id` (required) - The ID of a note +- `body` (required) - The content of a note diff --git a/doc/api/oauth2.md b/doc/api/oauth2.md new file mode 100644 index 00000000000..d416a826f79 --- /dev/null +++ b/doc/api/oauth2.md @@ -0,0 +1,102 @@ +# GitLab as an OAuth2 client + +This document is about using other OAuth authentication service providers to sign into GitLab. +If you want GitLab to be an OAuth authentication service provider to sign into other services please see the [Oauth2 provider documentation](../integration/oauth_provider.md). + +OAuth2 is a protocol that enables us to authenticate a user without requiring them to give their password. + +Before using the OAuth2 you should create an application in user's account. Each application gets a unique App ID and App Secret parameters. You should not share these. + +This functionality is based on [doorkeeper gem](https://github.com/doorkeeper-gem/doorkeeper) + +## Web Application Flow + +This flow is using for authentication from third-party web sites and is probably used the most. +It basically consists of an exchange of an authorization token for an access token. For more detailed info, check out the [RFC spec here](http://tools.ietf.org/html/rfc6749#section-4.1) + +This flow consists from 3 steps. + +### 1. Registering the client + +Create an application in user's account profile. + +### 2. Requesting authorization + +To request the authorization token, you should visit the `/oauth/authorize` endpoint. You can do that by visiting manually the URL: + +``` +http://localhost:3000/oauth/authorize?client_id=APP_ID&redirect_uri=REDIRECT_URI&response_type=code +``` + +Where REDIRECT_URI is the URL in your app where users will be sent after authorization. + +### 3. Requesting the access token + +To request the access token, you should use the returned code and exchange it for an access token. To do that you can use any HTTP client. In this case, I used rest-client: + +``` +parameters = 'client_id=APP_ID&client_secret=APP_SECRET&code=RETURNED_CODE&grant_type=AUTHORIZATION_CODE&redirect_uri=REDIRECT_URI' +RestClient.post 'http://localhost:3000/oauth/token', parameters + +# The response will be +{ + "access_token": "de6780bc506a0446309bd9362820ba8aed28aa506c71eedbe1c5c4f9dd350e54", + "token_type": "bearer", + "expires_in": 7200, + "refresh_token": "8257e65c97202ed1726cf9571600918f3bffb2544b26e00a61df9897668c33a1" +} +``` + +You can now make requests to the API with the access token returned. + +### Use the access token to access the API + +The access token allows you to make requests to the API on a behalf of a user. + +``` +GET https://localhost:3000/api/v3/user?access_token=OAUTH-TOKEN +``` + +Or you can put the token to the Authorization header: + +``` +curl -H "Authorization: Bearer OAUTH-TOKEN" https://localhost:3000/api/v3/user +``` + +## Resource Owner Password Credentials + +In this flow, a token is requested in exchange for the resource owner credentials (username and password). +The credentials should only be used when there is a high degree of trust between the resource owner and the client (e.g. the +client is part of the device operating system or a highly privileged application), and when other authorization grant types are not +available (such as an authorization code). + +Even though this grant type requires direct client access to the resource owner credentials, the resource owner credentials are used +for a single request and are exchanged for an access token. This grant type can eliminate the need for the client to store the +resource owner credentials for future use, by exchanging the credentials with a long-lived access token or refresh token. +You can do POST request to `/oauth/token` with parameters: + +``` +{ + "grant_type" : "password", + "username" : "user@example.com", + "password" : "sekret" +} +``` + +Then, you'll receive the access token back in the response: + +``` +{ + "access_token": "1f0af717251950dbd4d73154fdf0a474a5c5119adad999683f5b450c460726aa", + "token_type": "bearer", + "expires_in": 7200 +} +``` + +For testing you can use the oauth2 ruby gem: + +``` +client = OAuth2::Client.new('the_client_id', 'the_client_secret', :site => "http://example.com") +access_token = client.password.get_token('user@example.com', 'sekret') +puts access_token.token +``` diff --git a/doc/api/projects.md b/doc/api/projects.md index dfe3502b6e4..7fe244477db 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -1,5 +1,23 @@ # Projects + +### Project visibility level + +Project in GitLab has be either private, internal or public. +You can determine it by `visibility_level` field in project. + +Constants for project visibility levels are next: + +* Private. `visibility_level` is `0`. + Project access must be granted explicitly for each user. + +* Internal. `visibility_level` is `10`. + The project can be cloned by any logged in user. + +* Public. `visibility_level` is `20`. + The project can be cloned without any authentication. + + ## List projects Get a list of projects accessible by the authenticated user. @@ -11,6 +29,9 @@ GET /projects Parameters: - `archived` (optional) - if passed, limit by archived status +- `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at` +- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` +- `search` (optional) - Return list of authorized projects according to a search criteria ```json [ @@ -47,7 +68,8 @@ Parameters: "path": "diaspora", "updated_at": "2013-09-30T13: 46: 02Z" }, - "archived": false + "archived": false, + "avatar_url": "http://example.com/uploads/project/avatar/4/uploads/avatar.png" }, { "id": 6, @@ -82,7 +104,8 @@ Parameters: "path": "brightbox", "updated_at": "2013-09-30T13:46:02Z" }, - "archived": false + "archived": false, + "avatar_url": null } ] ``` @@ -95,6 +118,13 @@ Get a list of projects which are owned by the authenticated user. GET /projects/owned ``` +Parameters: + +- `archived` (optional) - if passed, limit by archived status +- `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at` +- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` +- `search` (optional) - Return list of authorized projects according to a search criteria + ### List ALL projects Get a list of all GitLab projects (admin only). @@ -103,6 +133,13 @@ Get a list of all GitLab projects (admin only). GET /projects/all ``` +Parameters: + +- `archived` (optional) - if passed, limit by archived status +- `order_by` (optional) - Return requests ordered by `id`, `name`, `path`, `created_at`, `updated_at` or `last_activity_at` fields. Default is `created_at` +- `sort` (optional) - Return requests sorted in `asc` or `desc` order. Default is `desc` +- `search` (optional) - Return list of authorized projects according to a search criteria + ### Get single project Get a specific project, identified by project ID or NAMESPACE/PROJECT_NAME, which is owned by the authenticated user. @@ -160,7 +197,8 @@ Parameters: "notification_level": 3 } }, - "archived": false + "archived": false, + "avatar_url": "http://example.com/uploads/project/avatar/3/uploads/avatar.png" } ``` @@ -186,6 +224,7 @@ Parameters: "target_id": 830, "target_type": "Issue", "author_id": 1, + "author_username": "john", "data": null, "target_title": "Public project search field" }, @@ -196,6 +235,7 @@ Parameters: "target_id": null, "target_type": null, "author_id": 1, + "author_username": "john", "data": { "before": "50d4420237a9de7be1304607147aec22e4a14af7", "after": "c5feabde2d8cd023215af4d2ceeb7a64839fc428", @@ -231,6 +271,7 @@ Parameters: "target_id": 840, "target_type": "Issue", "author_id": 1, + "author_username": "john", "data": null, "target_title": "Finish & merge Code search PR" } @@ -281,6 +322,31 @@ Parameters: - `visibility_level` (optional) - `import_url` (optional) +### Edit project + +Updates an existing project + +``` +PUT /projects/:id +``` + +Parameters: + +- `id` (required) - The ID of a project +- `name` (optional) - project name +- `path` (optional) - repository name for project +- `description` (optional) - short project description +- `default_branch` (optional) +- `issues_enabled` (optional) +- `merge_requests_enabled` (optional) +- `wiki_enabled` (optional) +- `snippets_enabled` (optional) +- `public` (optional) - if `true` same as setting visibility_level = 20 +- `visibility_level` (optional) + +On success, method returns 200 with the updated project. If parameters are +invalid, 400 is returned. + ### Fork project Forks a project into the user namespace of the authenticated user. @@ -510,7 +576,7 @@ Parameters: } ], "tree": "c68537c6534a02cc2b176ca1549f4ffa190b58ee", - "message": "give caolan credit where it's due (up top)", + "message": "give Caolan credit where it's due (up top)", "author": { "name": "Jeremy Ashkenas", "email": "jashkenas@example.com" @@ -625,6 +691,8 @@ GET /projects/search/:query Parameters: -- query (required) - A string contained in the project name -- per_page (optional) - number of projects to return per page -- page (optional) - the page to retrieve +- `query` (required) - A string contained in the project name +- `per_page` (optional) - number of projects to return per page +- `page` (optional) - the page to retrieve +- `order_by` (optional) - Return requests ordered by `id`, `name`, `created_at` or `last_activity_at` fields +- `sort` (optional) - Return requests sorted in `asc` or `desc` order diff --git a/doc/api/repositories.md b/doc/api/repositories.md index a412f60c0d9..33167453802 100644 --- a/doc/api/repositories.md +++ b/doc/api/repositories.md @@ -15,24 +15,21 @@ Parameters: ```json [ { - "name": "v1.0.0", "commit": { + "author_name": "John Smith", + "author_email": "john@example.com", + "authored_date": "2012-05-28T04:42:42-07:00", + "committed_date": "2012-05-28T04:42:42-07:00", + "committer_name": "Jack Smith", + "committer_email": "jack@example.com", "id": "2695effb5807a22ff3d138d593fd856244e155e7", - "parents": [], - "tree": "38017f2f189336fe4497e9d230c5bb1bf873f08d", "message": "Initial commit", - "author": { - "name": "John Smith", - "email": "john@example.com" - }, - "committer": { - "name": "Jack Smith", - "email": "jack@example.com" - }, - "authored_date": "2012-05-28T04:42:42-07:00", - "committed_date": "2012-05-28T04:42:42-07:00" + "parents_ids": [ + "2a4b78934375d7f53875269ffd4f45fd83a84ebe" + ] }, - "protected": null + "name": "v1.0.0", + "message": null } ] ``` @@ -53,24 +50,26 @@ Parameters: - `message` (optional) - Creates annotated tag. ```json -[ - { - "name": "v1.0.0", - "commit": { - "id": "2695effb5807a22ff3d138d593fd856244e155e7", - "parents": [], - "message": "Initial commit", - "authored_date": "2012-05-28T04:42:42-07:00", - "author_name": "John Smith", - "author email": "john@example.com", - "committer_name": "Jack Smith", - "committed_date": "2012-05-28T04:42:42-07:00", - "committer_email": "jack@example.com" - }, - "protected": false - } -] +{ + "commit": { + "author_name": "John Smith", + "author_email": "john@example.com", + "authored_date": "2012-05-28T04:42:42-07:00", + "committed_date": "2012-05-28T04:42:42-07:00", + "committer_name": "Jack Smith", + "committer_email": "jack@example.com", + "id": "2695effb5807a22ff3d138d593fd856244e155e7", + "message": "Initial commit", + "parents_ids": [ + "2a4b78934375d7f53875269ffd4f45fd83a84ebe" + ] + }, + "name": "v1.0.0", + "message": null +} ``` +The message will be `nil` when creating a lightweight tag otherwise +it will contain the annotation. It returns 200 if the operation succeed. In case of an error, 405 with an explaining error message is returned. diff --git a/doc/api/services.md b/doc/api/services.md new file mode 100644 index 00000000000..cbf767d1b25 --- /dev/null +++ b/doc/api/services.md @@ -0,0 +1,46 @@ +# Services + +## GitLab CI + +### Edit GitLab CI service + +Set GitLab CI service for a project. + +``` +PUT /projects/:id/services/gitlab-ci +``` + +Parameters: + +- `token` (required) - CI project token +- `project_url` (required) - CI project URL + +### Delete GitLab CI service + +Delete GitLab CI service settings for a project. + +``` +DELETE /projects/:id/services/gitlab-ci +``` + +## HipChat + +### Edit HipChat service + +Set HipChat service for project. + +``` +PUT /projects/:id/services/hipchat +``` +Parameters: + +- `token` (required) - HipChat token +- `room` (required) - HipChat room name + +### Delete HipChat service + +Delete HipChat service for a project. + +``` +DELETE /projects/:id/services/hipchat +``` diff --git a/doc/api/users.md b/doc/api/users.md index 3fdd3a75e88..a8b7685b503 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -78,7 +78,8 @@ GET /users "is_admin": false, "avatar_url": "http://localhost:3000/uploads/user/avatar/1/cd8.jpeg", "can_create_group": true, - "can_create_project": true + "can_create_project": true, + "projects_limit": 100 } ] ``` @@ -140,7 +141,8 @@ Parameters: "color_scheme_id": 2, "is_admin": false, "can_create_group": true, - "can_create_project": true + "can_create_project": true, + "projects_limit": 100 } ``` @@ -168,6 +170,7 @@ Parameters: - `bio` (optional) - User's biography - `admin` (optional) - User is admin - true or false (default) - `can_create_group` (optional) - User can create groups - true or false +- `confirm` (optional) - Require confirmation - true (default) or false ## User modification @@ -240,7 +243,8 @@ GET /user "color_scheme_id": 2, "is_admin": false, "can_create_group": true, - "can_create_project": true + "can_create_project": true, + "projects_limit": 100 } ``` @@ -257,12 +261,14 @@ GET /user/keys { "id": 1, "title": "Public key", - "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=" + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=", + "created_at": "2014-08-01T14:47:39.080Z" }, { "id": 3, "title": "Another Public key", - "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=" + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=", + "created_at": "2014-08-01T14:47:39.080Z" } ] ``` @@ -299,7 +305,8 @@ Parameters: { "id": 1, "title": "Public key", - "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=" + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=", + "created_at": "2014-08-01T14:47:39.080Z" } ``` @@ -316,6 +323,31 @@ Parameters: - `title` (required) - new SSH Key's title - `key` (required) - new SSH key +```json +{ + "created_at": "2015-01-21T17:44:33.512Z", + "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAMLrhYgI3atfrSD6KDas1b/3n6R/HP+bLaHHX6oh+L1vg31mdUqK0Ac/NjZoQunavoyzqdPYhFz9zzOezCrZKjuJDS3NRK9rspvjgM0xYR4d47oNZbdZbwkI4cTv/gcMlquRy0OvpfIvJtjtaJWMwTLtM5VhRusRuUlpH99UUVeXAAAAFQCVyX+92hBEjInEKL0v13c/egDCTQAAAIEAvFdWGq0ccOPbw4f/F8LpZqvWDydAcpXHV3thwb7WkFfppvm4SZte0zds1FJ+Hr8Xzzc5zMHe6J4Nlay/rP4ewmIW7iFKNBEYb/yWa+ceLrs+TfR672TaAgO6o7iSRofEq5YLdwgrwkMmIawa21FrZ2D9SPao/IwvENzk/xcHu7YAAACAQFXQH6HQnxOrw4dqf0NqeKy1tfIPxYYUZhPJfo9O0AmBW2S36pD2l14kS89fvz6Y1g8gN/FwFnRncMzlLY/hX70FSc/3hKBSbH6C6j8hwlgFKfizav21eS358JJz93leOakJZnGb8XlWvz1UJbwCsnR2VEY8Dz90uIk1l/UqHkA= loic@call", + "title": "ABC", + "id": 4 +} +``` + +Will return created key with status `201 Created` on success. If an +error occurs a `400 Bad Request` is returned with a message explaining the error: + +```json +{ + "message": { + "fingerprint": [ + "has already been taken" + ], + "key": [ + "has already been taken" + ] + } +} +``` + ## Add SSH key for user Create new key owned by specified user. Available only for admin diff --git a/doc/customization/libravatar.md b/doc/customization/libravatar.md new file mode 100644 index 00000000000..ee57fdc6590 --- /dev/null +++ b/doc/customization/libravatar.md @@ -0,0 +1,69 @@ +# Use Libravatar service with GitLab + +GitLab by default supports [Gravatar](gravatar.com) avatar service. +Libravatar is a service which delivers your avatar (profile picture) to other websites and their API is +[heavily based on gravatar](http://wiki.libravatar.org/api/). + +This means that it is not complicated to switch to Libravatar avatar service or even self hosted Libravatar server. + +# Configuration + +In [gitlab.yml gravatar section](https://gitlab.com/gitlab-org/gitlab-ce/blob/672bd3902d86b78d730cea809fce312ec49d39d7/config/gitlab.yml.example#L122) set +the configuration options as follows: + +## For HTTP + +```yml + gravatar: + enabled: true + # gravatar URLs: possible placeholders: %{hash} %{size} %{email} + plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" +``` + +## For HTTPS + +```yml + gravatar: + enabled: true + # gravatar URLs: possible placeholders: %{hash} %{size} %{email} + ssl_url: "https://seccdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" +``` + +## Self-hosted + +If you are [running your own libravatar service](http://wiki.libravatar.org/running_your_own/) the URL will be different in the configuration +but the important part is to provide the same placeholders so GitLab can parse the URL correctly. + +For example, you host a service on `http://libravatar.example.com` the `plain_url` you need to supply in `gitlab.yml` is + +`http://libravatar.example.com/avatar/%{hash}?s=%{size}&d=identicon` + + +## Omnibus-gitlab example + +In `/etc/gitlab/gitlab.rb`: + +#### For http + +```ruby +gitlab_rails['gravatar_enabled'] = true +gitlab_rails['gravatar_plain_url'] = "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" +``` + +#### For https + +```ruby +gitlab_rails['gravatar_enabled'] = true +gitlab_rails['gravatar_ssl_url'] = "https://seccdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" +``` + + +Run `sudo gitlab-ctl reconfigure` for changes to take effect. + + +## Default URL for missing images + +[Libravatar supports different sets](http://wiki.libravatar.org/api/) of `missing images` for emails not found on the Libravatar service. + +In order to use a different set other than `identicon`, replace `&d=identicon` portion of the URL with another supported set. +For example, you can use `retro` set in which case the URL would look like: `plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=retro"` diff --git a/doc/development/README.md b/doc/development/README.md index 20db6662aca..d5d264be19d 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -4,3 +4,5 @@ - [Shell commands](shell_commands.md) in the GitLab codebase - [Rake tasks](rake_tasks.md) for development - [CI setup](ci_setup.md) for testing GitLab +- [Sidekiq debugging](sidekiq_debugging.md) +- [UI guide](ui_guide.md) for building GitLab with existing css styles and elements diff --git a/doc/development/architecture.md b/doc/development/architecture.md index c4813d22eaa..714cc016004 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -8,6 +8,38 @@ EE releases are available not long after CE releases. To obtain the GitLab EE th Both EE and CE require an add-on component called gitlab-shell. It is obtained from the [gitlab-shell repository](https://gitlab.com/gitlab-org/gitlab-shell/tree/master). New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical. +## Physical office analogy + +You can imagine GitLab as a physical office. + +**The repositories** are the goods GitLab handling. +They can be stored in a warehouse. +This can be either a hard disk, or something more complex, such as a NFS filesystem; + +**Nginx** acts like the front-desk. +Users come to Nginx and request actions to be done by workers in the office; + +**The database** is a series of metal file cabinets with information on: + - The goods in the warehouse (metadata, issues, merge requests etc); + - The users coming to the front desk (permissions) + +**Redis** is a communication board with “cubby holes” that can contain tasks for office workers; + +**Sidekiq** is a worker that primarily handles sending out emails. +It takes tasks from the Redis communication board; + +**A Unicorn worker** is a worker that handles quick/mundane tasks. +They work with the communication board (Redis). +Their job description: + - check permissions by checking the user session stored in a Redis “cubby hole”; + - make tasks for Sidekiq; + - fetch stuff from the warehouse or move things around in there; + +**Gitlab-shell** is a third kind of worker that takes orders from a fax machine (SSH) instead of the front desk (HTTP). +Gitlab-shell communicates with Sidekiq via the “communication board” (Redis), and asks quick questions of the Unicorn workers either directly or via the front desk. + +**GitLab Enterprise Edition (the application)** is the collection of processes and business practices that the office is run by. + ## System Layout When referring to ~git in the pictures it means the home directory of the git user which is typically /home/git. @@ -38,7 +70,7 @@ To summarize here's the [directory structure of the `git` user home directory](. ps aux | grep '^git' -GitLab has several components to operate. As a system user (i.e. any user that is not the `git` user) it requires a persistent database (MySQL/PostreSQL) and redis database. It also uses Apache httpd or nginx to proxypass Unicorn. As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server running on port `8080` by default). Under the GitLab user there are normally 4 processes: `unicorn_rails master` (1 process), `unicorn_rails worker` (2 processes), `sidekiq` (1 process). +GitLab has several components to operate. As a system user (i.e. any user that is not the `git` user) it requires a persistent database (MySQL/PostreSQL) and redis database. It also uses Apache httpd or Nginx to proxypass Unicorn. As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server running on port `8080` by default). Under the GitLab user there are normally 4 processes: `unicorn_rails master` (1 process), `unicorn_rails worker` (2 processes), `sidekiq` (1 process). ### Repository access @@ -114,13 +146,13 @@ nginx Apache httpd -- [Explanation of apache logs](http://httpd.apache.org/docs/2.2/logs.html). +- [Explanation of Apache logs](http://httpd.apache.org/docs/2.2/logs.html). - `/var/log/apache2/` contains error and output logs (on Ubuntu). - `/var/log/httpd/` contains error and output logs (on RHEL). redis -- `/var/log/redis/redis.log` there are also logrotated logs there. +- `/var/log/redis/redis.log` there are also log-rotated logs there. PostgreSQL diff --git a/doc/development/ci_setup.md b/doc/development/ci_setup.md index b3e84183a41..f9b48868182 100644 --- a/doc/development/ci_setup.md +++ b/doc/development/ci_setup.md @@ -4,28 +4,30 @@ This document describes what services we use for testing GitLab and GitLab CI. We currently use three CI services to test GitLab: -1. GitLab CI on [GitHost.io](https://gitlab-ce.githost.io/projects/2/) for the [GitLab.com repo](https://gitlab.com/gitlab-org/gitlab-ce) +1. GitLab CI on [GitHost.io](https://gitlab-ce.githost.io/projects/4/) for the [GitLab.com repo](https://gitlab.com/gitlab-org/gitlab-ce) 2. GitLab CI at ci.gitlab.org to test the private GitLab B.V. repo at dev.gitlab.org 3. [Semephore](https://semaphoreapp.com/gitlabhq/gitlabhq/) for [GitHub.com repo](https://github.com/gitlabhq/gitlabhq) | Software @ configuration being tested | GitLab CI (ci.gitlab.org) | GitLab CI (GitHost.io) | Semaphore | -|---------------------------------------|---------------------------|------------------------|-----------| -| GitLab CE @ MySQL | ✓ | ✓ | | -| GitLab CE @ PostgreSQL | | | ✓ | -| GitLab EE @ MySQL | ✓ | | | -| GitLab CI @ MySQL | ✓ | | | -| GitLab CI @ PostgreSQL | | | ✓ | -| GitLab CI Runner | ✓ | | ✓ | -| GitLab Shell | ✓ | | ✓ | -| GitLab Shell | ✓ | | ✓ | +|---------------------------------------|---------------------------|---------------------------------------------------------------------------|-----------| +| GitLab CE @ MySQL | ✓ | ✓ [Core team can trigger builds](https://gitlab-ce.githost.io/projects/4) | | +| GitLab CE @ PostgreSQL | | | ✓ [Core team can trigger builds](https://semaphoreapp.com/gitlabhq/gitlabhq/branches/master) | +| GitLab EE @ MySQL | ✓ | | | +| GitLab CI @ MySQL | ✓ | | | +| GitLab CI @ PostgreSQL | | | ✓ | +| GitLab CI Runner | ✓ | | ✓ | +| GitLab Shell | ✓ | | ✓ | +| GitLab Shell | ✓ | | ✓ | + +Core team has access to trigger builds if needed for GitLab CE. We use [these build scripts](https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/examples/build_script_gitlab_ce.md) for testing with GitLab CI. # Build configuration on [Semaphore](https://semaphoreapp.com/gitlabhq/gitlabhq/) for testing the [GitHub.com repo](https://github.com/gitlabhq/gitlabhq) -Language: Ruby -Ruby verion: 2.1.2 -database.yml: pg +- Language: Ruby +- Ruby version: 2.1.2 +- database.yml: pg Build commands @@ -35,7 +37,10 @@ bundle install --deployment --path vendor/bundle (Setup) cp config/gitlab.yml.example config/gitlab.yml (Setup) bundle exec rake db:create (Setup) bundle exec rake spinach (Thread #1) -bundle exec rake spec (Thread #2) +bundle exec rake spec (thread #2) +bundle exec rake rubocop (thread #3) +bundle exec rake brakeman (thread #4) +bundle exec rake jasmine:ci (thread #5) ``` Use rubygems mirror. diff --git a/doc/development/omnibus.md b/doc/development/omnibus.md new file mode 100644 index 00000000000..0ba354d28a2 --- /dev/null +++ b/doc/development/omnibus.md @@ -0,0 +1,32 @@ +# What you should know about omnibus packages + +Most users install GitLab using our omnibus packages. As a developer it can be +good to know how the omnibus packages differ from what you have on your laptop +when you are coding. + +## Files are owned by root by default + +All the files in the Rails tree (`app/`, `config/` etc.) are owned by 'root' in +omnibus installations. This makes the installation simpler and it provides +extra security. The omnibus reconfigure script contains commands that give +write access to the 'git' user only where needed. + +For example, the 'git' user is allowed to write in the `log/` directory, in +`public/uploads`, and they are allowed to rewrite the `db/schema.rb` file. + +In other cases, the reconfigure script tricks GitLab into not trying to write a +file. For instance, GitLab will generate a `.secret` file if it cannot find one +and write it to the Rails root. In the omnibus packages, reconfigure writes the +`.secret` file first, so that GitLab never tries to write it. + +## Code, data and logs are in separate directories + +The omnibus design separates code (read-only, under `/opt/gitlab`) from data +(read/write, under `/var/opt/gitlab`) and logs (read/write, under +`/var/log/gitlab`). To make this happen the reconfigure script sets custom +paths where it can in GitLab config files, and where there are no path +settings, it uses symlinks. + +For example, `config/gitlab.yml` is treated as data so that file is a symlink. +The same goes for `public/uploads`. The `log/` directory is replaced by omnibus +with a symlink to `/var/log/gitlab/gitlab-rails`. diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 6d9ac161e91..53f8095cb13 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -1,6 +1,6 @@ # Rake tasks for developers -## Setup db with developer seeds: +## Setup db with developer seeds Note that if your db user does not have advanced privileges you must create the db manually before running this command. @@ -8,6 +8,10 @@ Note that if your db user does not have advanced privileges you must create the bundle exec rake setup ``` +The `setup` task is a alias for `gitlab:setup`. +This tasks calls `db:setup` to create the database, calls `add_limits_mysql` that adds limits to the database schema in case of a MySQL database and fianlly it calls `db:seed_fu` to seed the database. +Note: `db:setup` calls `db:seed` but this does nothing. + ## Run tests This runs all test suites present in GitLab. diff --git a/doc/development/shell_commands.md b/doc/development/shell_commands.md index 23c8365c340..821027f43fa 100644 --- a/doc/development/shell_commands.md +++ b/doc/development/shell_commands.md @@ -1,5 +1,8 @@ # Guidelines for shell commands in the GitLab codebase +This document contains guidelines for working with processes and files in the GitLab codebase. +These guidelines are meant to make your code more reliable _and_ secure. + ## References - [Google Ruby Security Reviewer's Guide](https://code.google.com/p/ruby-security/wiki/Guide) @@ -105,7 +108,72 @@ In other repositories, such as gitlab-shell you can also use `IO.popen`. ```ruby # Safe IO.popen example -logs = IO.popen(%W(git log), chdir: repo_dir).read +logs = IO.popen(%W(git log), chdir: repo_dir) { |p| p.read } ``` Note that unlike `Gitlab::Popen.popen`, `IO.popen` does not capture standard error. + +## Avoid user input at the start of path strings + +Various methods for opening and reading files in Ruby can be used to read the +standard output of a process instead of a file. The following two commands do +roughly the same: + +``` +`touch /tmp/pawned-by-backticks` +File.read('|touch /tmp/pawned-by-file-read') +``` + +The key is to open a 'file' whose name starts with a `|`. +Affected methods include Kernel#open, File::read, File::open, IO::open and IO::read. + +You can protect against this behavior of 'open' and 'read' by ensuring that an +attacker cannot control the start of the filename string you are opening. For +instance, the following is sufficient to protect against accidentally starting +a shell command with `|`: + +``` +# we assume repo_path is not controlled by the attacker (user) +path = File.join(repo_path, user_input) +# path cannot start with '|' now. +File.read(path) +``` + +If you have to use user input a relative path, prefix `./` to the path. + +Prefixing user-supplied paths also offers extra protection against paths +starting with `-` (see the discussion about using `--` above). + +## Guard against path traversal + +Path traversal is a security where the program (GitLab) tries to restrict user +access to a certain directory on disk, but the user manages to open a file +outside that directory by taking advantage of the `../` path notation. + +``` +# Suppose the user gave us a path and they are trying to trick us +user_input = '../other-repo.git/other-file' + +# We look up the repo path somewhere +repo_path = 'repositories/user-repo.git' + +# The intention of the code below is to open a file under repo_path, but +# because the user used '..' she can 'break out' into +# 'repositories/other-repo.git' +full_path = File.join(repo_path, user_input) +File.open(full_path) do # Oops! +``` + +A good way to protect against this is to compare the full path with its +'absolute path' according to Ruby's `File.absolute_path`. + +``` +full_path = File.join(repo_path, user_input) +if full_path != File.absolute_path(full_path) + raise "Invalid path: #{full_path.inspect}" +end + +File.open(full_path) do # Etc. +``` + +A check like this could have avoided CVE-2013-4583. diff --git a/doc/development/sidekiq_debugging.md b/doc/development/sidekiq_debugging.md new file mode 100644 index 00000000000..cea11e5f126 --- /dev/null +++ b/doc/development/sidekiq_debugging.md @@ -0,0 +1,14 @@ +# Sidekiq debugging + +## Log arguments to Sidekiq jobs + +If you want to see what arguments are being passed to Sidekiq jobs you can set +the SIDEKIQ_LOG_ARGUMENTS environment variable. + +``` +SIDEKIQ_LOG_ARGUMENTS=1 bundle exec foreman start +``` + +It is not recommend to enable this setting in production because some Sidekiq +jobs (such as sending a password reset email) take secret arguments (for +example the password reset token). diff --git a/doc/development/ui_guide.md b/doc/development/ui_guide.md new file mode 100644 index 00000000000..2f01defc11d --- /dev/null +++ b/doc/development/ui_guide.md @@ -0,0 +1,12 @@ +# UI Guide for building GitLab + +## Best practices for creating new pages in GitLab + +TODO: write some best practices when develop GitLab features. + +## GitLab UI development kit + +We created a page inside GitLab where you can check commonly used html and css elements. + +When you run GitLab instance locally - just visit http://localhost:3000/help/ui page to see UI examples +you can use during GitLab development. diff --git a/doc/hooks/custom_hooks.md b/doc/hooks/custom_hooks.md new file mode 100644 index 00000000000..f7d4f3de68b --- /dev/null +++ b/doc/hooks/custom_hooks.md @@ -0,0 +1,41 @@ +# Custom Git Hooks + +**Note: Custom git hooks must be configured on the filesystem of the GitLab +server. Only GitLab server administrators will be able to complete these tasks. +Please explore webhooks as an option if you do not have filesystem access.** + +Git natively supports hooks that are executed on different actions. +Examples of server-side git hooks include pre-receive, post-receive, and update. +See +[Git SCM Server-Side Hooks](http://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#Server-Side-Hooks) +for more information about each hook type. + +As of gitlab-shell version 2.2.0 (which requires GitLab 7.5+), GitLab +administrators can add custom git hooks to any GitLab project. + +## Setup + +Normally, git hooks are placed in the repository or project's `hooks` directory. +GitLab creates a symlink from each project's `hooks` directory to the +gitlab-shell `hooks` directory for ease of maintenance between gitlab-shell +upgrades. As such, custom hooks are implemented a little differently. Behavior +is exactly the same once the hook is created, though. Follow these steps to +set up a custom hook. + +1. Pick a project that needs a custom git hook. +1. On the GitLab server, navigate to the project's repository directory. +For an installation from source the path is usually +`/home/git/repositories/<group>/<project>.git`. For Omnibus installs the path is +usually `/var/opt/gitlab/git-data/repositories/<group>/<project>.git`. +1. Create a new directory in this location called `custom_hooks`. +1. Inside the new `custom_hooks` directory, create a file with a name matching +the hook type. For a pre-receive hook the file name should be `pre-receive` with +no extension. +1. Make the hook file executable and make sure it's owned by git. +1. Write the code to make the git hook function as expected. Hooks can be +in any language. Ensure the 'shebang' at the top properly reflects the language +type. For example, if the script is in Ruby the shebang will probably be +`#!/usr/bin/env ruby`. + +That's it! Assuming the hook code is properly implemented the hook will fire +as appropriate. diff --git a/doc/install/installation.md b/doc/install/installation.md index 0d1a8da4d1b..d6208bb0797 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -1,10 +1,14 @@ -# Installation +# Installation from source -## Select Version to Install +## Consider the Omnibus package installation + +Since an installation from source is a lot of work and error prone we strongly recommend the fast and reliable [Omnibus package installation](https://about.gitlab.com/downloads/) (deb/rpm). -Make sure you view [this installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md) from the branch (version) of GitLab you would like to install. In most cases this should be the highest numbered stable branch (example shown below). +## Select Version to Install - +Make sure you view [this installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md) from the tag (version) of GitLab you would like to install. +In most cases this should be the highest numbered production tag (without rc in it). +You can select the tag in the version dropdown in the top left corner of GitLab (below the menu bar). If the highest number stable branch is unclear please check the [GitLab Blog](https://about.gitlab.com/blog/) for installation guide links by version. @@ -18,7 +22,9 @@ This is the official installation guide to set up a production server. To set up The following steps have been known to work. Please **use caution when you deviate** from this guide. Make sure you don't violate any assumptions GitLab makes about its environment. For example many people run into permission problems because they changed the location of directories or run services as the wrong user. -If you find a bug/error in this guide please **submit a merge request** following the [contributing guide](../../CONTRIBUTING.md). +If you find a bug/error in this guide please **submit a merge request** +following the +[contributing guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md). ## Overview @@ -50,7 +56,7 @@ up-to-date and install it. Install the required packages (needed to compile Ruby and native extensions to Ruby gems): - sudo apt-get install -y build-essential zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libreadline-dev libncurses5-dev libffi-dev curl openssh-server redis-server checkinstall libxml2-dev libxslt-dev libcurl4-openssl-dev libicu-dev logrotate python-docutils pkg-config cmake + sudo apt-get install -y build-essential zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libreadline-dev libncurses5-dev libffi-dev curl openssh-server redis-server checkinstall libxml2-dev libxslt-dev libcurl4-openssl-dev libicu-dev logrotate python-docutils pkg-config cmake libkrb5-dev nodejs Make sure you have the right version of Git installed @@ -70,8 +76,9 @@ Is the system packaged Git too old? Remove it and compile from source. # Download and compile from source cd /tmp - curl -L --progress https://www.kernel.org/pub/software/scm/git/git-2.0.0.tar.gz | tar xz - cd git-2.0.0/ + curl -L --progress https://www.kernel.org/pub/software/scm/git/git-2.1.2.tar.gz | tar xz + cd git-2.1.2/ + ./configure make prefix=/usr/local all # Install into /usr/local/bin @@ -87,7 +94,7 @@ Then select 'Internet Site' and press enter to confirm the hostname. ## 2. Ruby -The use of ruby version managers such as [RVM](http://rvm.io/), [rbenv](https://github.com/sstephenson/rbenv) or [chruby](https://github.com/postmodern/chruby) with GitLab in production frequently leads to hard to diagnose problems. For example, GitLab Shell is called from OpenSSH and having a version manager can prevent pushing and pulling over SSH. Version managers are not supported and we strongly advise everyone to follow the instructions below to use a system ruby. +The use of Ruby version managers such as [RVM](http://rvm.io/), [rbenv](https://github.com/sstephenson/rbenv) or [chruby](https://github.com/postmodern/chruby) with GitLab in production frequently leads to hard to diagnose problems. For example, GitLab Shell is called from OpenSSH and having a version manager can prevent pushing and pulling over SSH. Version managers are not supported and we strongly advise everyone to follow the instructions below to use a system Ruby. Remove the old Ruby 1.8 if present @@ -96,8 +103,8 @@ Remove the old Ruby 1.8 if present Download Ruby and compile it: mkdir /tmp/ruby && cd /tmp/ruby - curl -L --progress ftp://ftp.ruby-lang.org/pub/ruby/2.1/ruby-2.1.2.tar.gz | tar xz - cd ruby-2.1.2 + curl -L --progress http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.5.tar.gz | tar xz + cd ruby-2.1.5 ./configure --disable-install-rdoc make sudo make install @@ -122,7 +129,8 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da # Login to PostgreSQL sudo -u postgres psql -d template1 - # Create a user for GitLab. + # Create a user for GitLab + # Do not type the 'template1=#', this is part of the prompt template1=# CREATE USER git CREATEDB; # Create the GitLab production database & grant all privileges on database @@ -134,6 +142,9 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da # Try connecting to the new database with the new user sudo -u git -H psql -d gitlabhq_production + # Quit the database session + gitlabhq_production> \q + ## 5. Redis sudo apt-get install redis-server @@ -146,6 +157,17 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da # Enable Redis socket for default Debian / Ubuntu path echo 'unixsocket /var/run/redis/redis.sock' | sudo tee -a /etc/redis/redis.conf + # Grant permission to the socket to all members of the redis group + echo 'unixsocketperm 770' | sudo tee -a /etc/redis/redis.conf + + # Create the directory which contains the socket + mkdir /var/run/redis + chown redis:redis /var/run/redis + chmod 755 /var/run/redis + # Persist the directory which contains the socket, if applicable + if [ -d /etc/tmpfiles.d ]; then + echo 'd /var/run/redis 0755 redis redis 10d -' | sudo tee -a /etc/tmpfiles.d/redis.conf + fi # Activate the changes to redis.conf sudo service redis-server restart @@ -161,9 +183,9 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da ### Clone the Source # Clone GitLab repository - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 7-3-stable gitlab + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 7-9-stable gitlab -**Note:** You can change `7-3-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! +**Note:** You can change `7-9-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! ### Configure It @@ -179,7 +201,7 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da # Make sure GitLab can write to the log/ and tmp/ directories sudo chown -R git log/ sudo chown -R git tmp/ - sudo chmod -R u+rwX log/ + sudo chmod -R u+rwX,go-w log/ sudo chmod -R u+rwX tmp/ # Create directory for satellites @@ -258,7 +280,7 @@ We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](da GitLab Shell is an SSH access and repository management software developed specially for GitLab. # Run the installation task for gitlab-shell (replace `REDIS_URL` if needed): - sudo -u git -H bundle exec rake gitlab:shell:install[v2.0.1] REDIS_URL=unix:/var/run/redis/redis.sock RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:shell:install[v2.6.0] REDIS_URL=unix:/var/run/redis/redis.sock RAILS_ENV=production # By default, the gitlab-shell config is generated from your main GitLab config. # You can review (and modify) the gitlab-shell config as follows: @@ -274,9 +296,9 @@ GitLab Shell is an SSH access and repository management software developed speci # When done you see 'Administrator account created:' -**Note:** You can set the Administrator password by supplying it in environmental variable `GITLAB_ROOT_PASSWORD`, eg.: +**Note:** You can set the Administrator/root password by supplying it in environmental variable `GITLAB_ROOT_PASSWORD` as seen below. If you don't set the password (and it is set to the default one) please wait with exposing GitLab to the public internet until the installation is done and you've logged into the server the first time. During the first login you'll be forced to change the default password. - sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production GITLAB_ROOT_PASSWORD=newpassword + sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production GITLAB_ROOT_PASSWORD=yourpassword ### Install Init Script @@ -363,15 +385,17 @@ NOTE: Supply `SANITIZE=true` environment variable to `gitlab:check` to omit proj ### Initial Login -Visit YOUR_SERVER in your web browser for your first GitLab login. The setup has created an admin account for you. You can use it to log in: +Visit YOUR_SERVER in your web browser for your first GitLab login. The setup has created a default admin account for you. You can use it to log in: root 5iveL!fe -**Important Note:** Please go over to your profile page and immediately change the password, so nobody can access your GitLab by using this login information later on. +**Important Note:** On login you'll be prompted to change the password. **Enjoy!** +You can use `sudo service gitlab start` and `sudo service gitlab stop` to start and stop GitLab. + ## Advanced Setup Tips ### Using HTTPS @@ -435,4 +459,4 @@ You can configure LDAP authentication in `config/gitlab.yml`. Please restart Git ### Using Custom Omniauth Providers -See the [omniauth integration document](doc/integration/omniauth.md) +See the [omniauth integration document](../integration/omniauth.md) diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 3e4c6a28c0e..f42af65796f 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -7,7 +7,7 @@ - Ubuntu - Debian - CentOS -- RedHat Enterprise Linux (please use the CentOS packages and instructions) +- Red Hat Enterprise Linux (please use the CentOS packages and instructions) - Scientific Linux (please use the CentOS packages and instructions) - Oracle Linux (please use the CentOS packages and instructions) @@ -22,9 +22,9 @@ For the installations options please see [the installation page on the GitLab we - FreeBSD On the above unsupported distributions is still possible to install GitLab yourself. -Please see the [manual installation guide](https://github.com/gitlabhq/gitlabhq/blob/master/doc/install/installation.md) and the [unofficial installation guides](https://github.com/gitlabhq/gitlab-public-wiki/wiki/Unofficial-Installation-Guides) on the public wiki for more information. +Please see the [installation from source guide](https://github.com/gitlabhq/gitlabhq/blob/master/doc/install/installation.md) and the [unofficial installation guides](https://github.com/gitlabhq/gitlab-public-wiki/wiki/Unofficial-Installation-Guides) on the public wiki for more information. -### Non Unix operating systems such as Windows +### Non-Unix operating systems such as Windows GitLab is developed for Unix operating systems. GitLab does **not** run on Windows and we have no plans of supporting it in the near future. @@ -38,6 +38,16 @@ We love [JRuby](http://jruby.org/) and [Rubinius](http://rubini.us/) but GitLab ## Hardware requirements +### Storage + +The necessary hard drive space largely depends on the size of the repos you want to store in GitLab but as a *rule of thumb* you should have at least twice as much free space as all your repos combined take up. You need twice the storage because [GitLab satellites](structure.md) contain an extra copy of each repo. + +If you want to be flexible about growing your hard drive space in the future consider mounting it using LVM so you can add more hard drives when you need them. + +Apart from a local hard drive you can also mount a volume that supports the network file system (NFS) protocol. This volume might be located on a file server, a network attached storage (NAS) device, a storage area network (SAN) or on an Amazon Web Services (AWS) Elastic Block Store (EBS) volume. + +If you have enough RAM memory and a recent CPU the speed of GitLab is mainly limited by hard drive seek times. Having a fast drive (7200 RPM and up) or a solid state drive (SSD) will improve the responsiveness of GitLab. + ### CPU - 1 core works supports up to 100 users but the application can be a bit slower due to having all workers and background jobs running on the same core @@ -50,11 +60,10 @@ We love [JRuby](http://jruby.org/) and [Rubinius](http://rubini.us/) but GitLab ### Memory -- 512MB is the absolute minimum but we do not recommend this amount of memory. -You will either need to configure 512MB or 1.5GB of swap space. -With 512MB of swap space you must configure only one unicorn worker. -With one unicorn worker only git over ssh access will work because the git over http access requires two running workers (one worker to receive the user request and one worker for the authorization check). -If you use SSD storage and configure 1.5GB of swap space you can use two Unicorn workers, this will allow http access but it will still be slow. +You need at least 2GB of addressable memory (RAM + swap) to install and use GitLab! +With less memory GitLab will give strange errors during the reconfigure run and 500 errors during usage. + +- 512MB RAM + 1.5GB of swap is the absolute minimum but we strongly **advise against** this amount of memory. See the unicorn worker section below for more advise. - 1GB RAM + 1GB swap supports up to 100 users - **2GB RAM** is the **recommended** memory size and supports up to 500 users - 4GB RAM supports up to 2,000 users @@ -65,15 +74,19 @@ If you use SSD storage and configure 1.5GB of swap space you can use two Unicorn Notice: The 25 workers of Sidekiq will show up as separate processes in your process overview (such as top or htop) but they share the same RAM allocation since Sidekiq is a multithreaded application. -### Storage +## Unicorn Workers -The necessary hard drive space largely depends on the size of the repos you want to store in GitLab. But as a *rule of thumb* you should have at least twice as much free space as your all repos combined take up. You need twice the storage because [GitLab satellites](structure.md) contain an extra copy of each repo. +It's possible to increase the amount of unicorn workers and tis will usually help for to reduce the response time of the applications and increase the ability to handle parallel requests. -If you want to be flexible about growing your hard drive space in the future consider mounting it using LVM so you can add more hard drives when you need them. +For most instances we recommend using: CPU cores + 1 = unicorn workers. +So for a machine with 2 cores, 3 unicorn workers is ideal. -Apart from a local hard drive you can also mount a volume that supports the network file system (NFS) protocol. This volume might be located on a file server, a network attached storage (NAS) device, a storage area network (SAN) or on an Amazon Web Services (AWS) Elastic Block Store (EBS) volume. +For all machines that have 1GB and up we recommend a minimum of three unicorn workers. +If you have a 512MB machine with a magnetic (non-SSD) swap drive we recommend to configure only one Unicorn worker to prevent excessive swapping. +With one Unicorn worker only git over ssh access will work because the git over HTTP access requires two running workers (one worker to receive the user request and one worker for the authorization check). +If you have a 512MB machine with a SSD drive you can use two Unicorn workers, this will allow HTTP access although it will be slow due to swapping. -If you have enough RAM memory and a recent CPU the speed of GitLab is mainly limited by hard drive seek times. Having a fast drive (7200 RPM and up) or a solid state drive (SSD) will improve the responsiveness of GitLab. +To change the Unicorn workers when you have the Omnibus package please see [the Unicorn settings in the Omnibus GitLab documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md#unicorn-settings). ## Database @@ -90,7 +103,7 @@ On a very active server (10,000 active users) the Sidekiq process can use 1GB+ o ## Supported web browsers - Chrome (Latest stable version) -- Firefox (Latest released version) +- Firefox (Latest released version and [latest ESR version](https://www.mozilla.org/en-US/firefox/organizations/)) - Safari 7+ (known problem: required fields in html5 do not work) - Opera (Latest released version) - IE 10+ diff --git a/doc/integration/README.md b/doc/integration/README.md index 357ed038314..286bd34a0bd 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -6,14 +6,16 @@ See the documentation below for details on how to configure these services. - [External issue tracker](external-issue-tracker.md) Redmine, JIRA, etc. - [LDAP](ldap.md) Set up sign in via LDAP -- [OmniAuth](omniauth.md) Sign in via Twitter, GitHub, and Google via OAuth. +- [OmniAuth](omniauth.md) Sign in via Twitter, GitHub, GitLab, and Google via OAuth. - [Slack](slack.md) Integrate with the Slack chat service +- [OAuth2 provider](oauth_provider.md) OAuth2 application creation +- [Gmail](gitlab_buttons_in_gmail.md) Adds GitLab actions to messages -Jenkins support is [available in GitLab EE](http://doc.gitlab.com/ee/integration/jenkins.html). +GitLab Enterprise Edition contains [advanced JIRA support](http://doc.gitlab.com/ee/integration/jira.html) and [advanced Jenkins support](http://doc.gitlab.com/ee/integration/jenkins.html). ## Project services -Integration with services such as Campfire, Flowdock, Gemnasium, HipChat, PivotalTracker and Slack are available in the from of a Project Service. +Integration with services such as Campfire, Flowdock, Gemnasium, HipChat, Pivotal Tracker, and Slack are available in the form of a Project Service. You can find these within GitLab in the Services page under Project Settings if you are at least a master on the project. Project Services are a bit like plugins in that they allow a lot of freedom in adding functionality to GitLab, for example there is also a service that can send an email every time someone pushes new commits. Because GitLab is open source we can ship with the code and tests for all plugins. diff --git a/doc/integration/bitbucket.md b/doc/integration/bitbucket.md new file mode 100644 index 00000000000..cc6389f5aaf --- /dev/null +++ b/doc/integration/bitbucket.md @@ -0,0 +1,121 @@ +# Integrate your server with Bitbucket + +Import projects from Bitbucket and login to your GitLab instance with your Bitbucket account. + +To enable the Bitbucket OmniAuth provider you must register your application with Bitbucket. +Bitbucket will generate an application ID and secret key for you to use. + +1. Sign in to Bitbucket. + +1. Navigate to your individual user settings or a team's settings, depending on how you want the application registered. It does not matter if the application is registered as an individual or a team - that is entirely up to you. + +1. Select "OAuth" in the left menu. + +1. Select "Add consumer". + +1. Provide the required details. + - Name: This can be anything. Consider something like "\<Organization\>'s GitLab" or "\<Your Name\>'s GitLab" or something else descriptive. + - Application description: Fill this in if you wish. + - URL: The URL to your GitLab installation. 'https://gitlab.company.com' +1. Select "Save". + +1. You should now see a Key and Secret in the list of OAuth customers. + Keep this page open as you continue configuration. + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For instalations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "bitbucket", + "app_id" => "YOUR_KEY", + "app_secret" => "YOUR_APP_SECRET", + "url" => "https://bitbucket.org/" + } + ] + ``` + + For installation from source: + + ``` + - { name: 'bitbucket', app_id: 'YOUR_KEY', + app_secret: 'YOUR_APP_SECRET' } + ``` + +1. Change 'YOUR_APP_ID' to the key from the Bitbucket application page from step 7. + +1. Change 'YOUR_APP_SECRET' to the secret from the Bitbucket application page from step 7. + +1. Save the configuration file. + +1. Restart GitLab for the changes to take effect. + +On the sign in page there should now be a Bitbucket icon below the regular sign in form. +Click the icon to begin the authentication process. Bitbucket will ask the user to sign in and authorize the GitLab application. +If everything goes well the user will be returned to GitLab and will be signed in. + +## Bitbucket project import + +To allow projects to be imported directly into GitLab, Bitbucket requires two extra setup steps compared to GitHub and GitLab.com. + +Bitbucket doesn't allow OAuth applications to clone repositories over HTTPS, and instead requires GitLab to use SSH and identify itself using your GitLab server's SSH key. + +### Step 1: Known hosts + +To allow GitLab to connect to Bitbucket over SSH, you need to add 'bitbucket.org' to your GitLab server's known SSH hosts. Take the following steps to do so: + +1. Manually connect to 'bitbucket.org' over SSH, while logged in as the `git` account that GitLab will use: + + ```sh + ssh git@bitbucket.org + ``` + +1. Verify the RSA key fingerprint you'll see in the response matches the one in the [Bitbucket documentation](https://confluence.atlassian.com/display/BITBUCKET/Use+the+SSH+protocol+with+Bitbucket#UsetheSSHprotocolwithBitbucket-KnownhostorBitbucket'spublickeyfingerprints) (the specific IP address doesn't matter): + + ```sh + The authenticity of host 'bitbucket.org (207.223.240.182)' can't be established. + RSA key fingerprint is 97:8c:1b:f2:6f:14:6b:5c:3b:ec:aa:46:46:74:7c:40. + Are you sure you want to continue connecting (yes/no)? + ``` + +1. If the fingerprint matches, type `yes` to continue connecting and have 'bitbucket.org' be added to your known hosts. + +1. Your GitLab server is now able to connect to Bitbucket over SSH. Continue to step 2: + +### Step 2: Public key + +To be able to access repositories on Bitbucket, GitLab will automatically register your public key with Bitbucket as a deploy key for the repositories to be imported. Your public key needs to be at `~/.ssh/id_rsa.pub`, which will expand to `/home/git/.ssh/id_rsa.pub` in most configurations. + +If you have that file in place, you're all set and should see the "Import projects from Bitbucket" option enabled. If you don't, do the following: + +1. Create a new SSH key: + + ```sh + sudo -u git -H ssh-keygen + ``` + + Make sure to use an **empty passphrase**. + +2. Restart GitLab to allow it to find the new public key. + +You should now see the "Import projects from Bitbucket" option on the New Project page enabled. diff --git a/doc/integration/external-issue-tracker.md b/doc/integration/external-issue-tracker.md index 87af94512ed..96755707dee 100644 --- a/doc/integration/external-issue-tracker.md +++ b/doc/integration/external-issue-tracker.md @@ -1,13 +1,39 @@ # External issue tracker -GitLab has a great issue tracker but you can also use an external issue tracker such as JIRA, Bugzilla or Redmine. This is something that you can turn on per GitLab project. If for example you configure JIRA it provides the following functionality: +GitLab has a great issue tracker but you can also use an external issue tracker such as Jira, Bugzilla or Redmine. You can configure issue trackers per GitLab project. For instance, if you configure Jira it allows you to do the following: -- the 'Issues' link on the GitLab project pages takes you to the appropriate JIRA issue index; -- clicking 'New issue' on the project dashboard creates a new JIRA issue; -- To reference JIRA issue PROJECT-1234 in comments, use syntax PROJECT-1234. Commit messages get turned into HTML links to the corresponding JIRA issue. +- the 'Issues' link on the GitLab project pages takes you to the appropriate Jira issue index; +- clicking 'New issue' on the project dashboard creates a new Jira issue; +- To reference Jira issue PROJECT-1234 in comments, use syntax PROJECT-1234. Commit messages get turned into HTML links to the corresponding Jira issue. - + -You can configure the integration in the gitlab.yml configuration file. +GitLab Enterprise Edition contains [advanced JIRA support](http://doc.gitlab.com/ee/integration/jira.html). + +## Configuration + +### Project Service + +You can enable an external issue tracker per project. As an example, we will configure `Redmine` for project named gitlab-ci. + +Fill in the required details on the page: + + + +* `description` A name for the issue tracker (to differentiate between instances, for example). +* `project_url` The URL to the project in Redmine which is being linked to this GitLab project. +* `issues_url` The URL to the issue in Redmine project that is linked to this GitLab project. Note that the `issues_url` requires `:id` in the url. This id is used by GitLab as a placeholder to replace the issue number. +* `new_issue_url` This is the URL to create a new issue in Redmine for the project linked to this GitLab project. + +### Service Template + +It is necessary to configure the external issue tracker per project, because project specific details are needed for the integration with GitLab. +The admin can add a service template that sets a default for each project. This makes it much easier to configure individual projects. + +In GitLab Admin section, navigate to `Service Templates` and choose the service template you want to create: + + + +After the template is created, the template details will be pre-filled on the project service page. Support to add your commits to the Jira ticket automatically is [available in GitLab EE](http://doc.gitlab.com/ee/integration/jira.html). diff --git a/doc/integration/github.md b/doc/integration/github.md index 714593d8266..b64501c2aaa 100644 --- a/doc/integration/github.md +++ b/doc/integration/github.md @@ -1,6 +1,9 @@ -# GitHub OAuth2 OmniAuth Provider +# Integrate your server with GitHub -To enable the GitHub OmniAuth provider you must register your application with GitHub. GitHub will generate a client ID and secret key for you to use. +Import projects from GitHub and login to your GitLab instance with your GitHub account. + +To enable the GitHub OmniAuth provider you must register your application with GitHub. +GitHub will generate an application ID and secret key for you to use. 1. Sign in to GitHub. @@ -14,35 +17,63 @@ To enable the GitHub OmniAuth provider you must register your application with G - Application name: This can be anything. Consider something like "\<Organization\>'s GitLab" or "\<Your Name\>'s GitLab" or something else descriptive. - Homepage URL: The URL to your GitLab installation. 'https://gitlab.company.com' - Application description: Fill this in if you wish. - - Authorization callback URL: 'https://gitlab.company.com/users/auth/github/callback' + - Authorization callback URL: 'https://gitlab.company.com/' 1. Select "Register application". -1. You should now see a Client ID and Client Secret near the top right of the page (see screenshot). Keep this page open as you continue configuration.  +1. You should now see a Client ID and Client Secret near the top right of the page (see screenshot). + Keep this page open as you continue configuration. +  1. On your GitLab server, open the configuration file. + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For instalations from source: + ```sh - cd /home/git/gitlab + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml + sudo -u git -H editor config/gitlab.yml ``` -1. Find the section dealing with OmniAuth. See [Initial OmniAuth Configuration](README.md#initial-omniauth-configuration) for more details. +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "github", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "url" => "https://github.com/", + "args" => { "scope" => "user:email" } + } + ] + ``` -1. Under `providers:` uncomment (or add) lines that look like the following: + For installation from source: ``` - - { name: 'github', app_id: 'YOUR APP ID', - app_secret: 'YOUR APP SECRET', - args: { scope: 'user:email' } } + - { name: 'github', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { scope: 'user:email' } } ``` -1. Change 'YOUR APP ID' to the client ID from the GitHub application page from step 7. +1. Change 'YOUR_APP_ID' to the client ID from the GitHub application page from step 7. -1. Change 'YOUR APP SECRET' to the client secret from the GitHub application page from step 7. +1. Change 'YOUR_APP_SECRET' to the client secret from the GitHub application page from step 7. 1. Save the configuration file. 1. Restart GitLab for the changes to take effect. -On the sign in page there should now be a GitHub icon below the regular sign in form. Click the icon to begin the authentication process. GitHub will ask the user to sign in and authorize the GitLab application. If everything goes well the user will be returned to GitLab and will be signed in. +On the sign in page there should now be a GitHub icon below the regular sign in form. +Click the icon to begin the authentication process. GitHub will ask the user to sign in and authorize the GitLab application. +If everything goes well the user will be returned to GitLab and will be signed in. diff --git a/doc/integration/github_app.png b/doc/integration/github_app.png Binary files differindex c0873b2e20d..d890345ced9 100644 --- a/doc/integration/github_app.png +++ b/doc/integration/github_app.png diff --git a/doc/integration/gitlab.md b/doc/integration/gitlab.md new file mode 100644 index 00000000000..216f1f11a9b --- /dev/null +++ b/doc/integration/gitlab.md @@ -0,0 +1,84 @@ +# Integrate your server with GitLab.com + +Import projects from GitLab.com and login to your GitLab instance with your GitLab.com account. + +To enable the GitLab.com OmniAuth provider you must register your application with GitLab.com. +GitLab.com will generate an application ID and secret key for you to use. + +1. Sign in to GitLab.com + +1. Navigate to your profile settings. + +1. Select "Applications" in the left menu. + +1. Select "New application". + +1. Provide the required details. + - Name: This can be anything. Consider something like "\<Organization\>'s GitLab" or "\<Your Name\>'s GitLab" or something else descriptive. + - Redirect URI: + + ``` + http://your-gitlab.example.com/import/gitlab/callback + http://your-gitlab.example.com/users/auth/gitlab/callback + ``` + + The first link is required for the importer and second for the authorization. + +1. Select "Submit". + +1. You should now see a Client ID and Client Secret near the top right of the page (see screenshot). + Keep this page open as you continue configuration. +  + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For instalations from source: + + ```sh + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "gitlab", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "args" => { "scope" => "api" } + } + ] + ``` + + For installations from source: + + ``` + - { name: 'gitlab', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { scope: 'api' } } + ``` + +1. Change 'YOUR_APP_ID' to the Application ID from the GitLab.com application page. + +1. Change 'YOUR_APP_SECRET' to the secret from the GitLab.com application page. + +1. Save the configuration file. + +1. Restart GitLab for the changes to take effect. + +On the sign in page there should now be a GitLab.com icon below the regular sign in form. +Click the icon to begin the authentication process. GitLab.com will ask the user to sign in and authorize the GitLab application. +If everything goes well the user will be returned to your GitLab instance and will be signed in. diff --git a/doc/integration/gitlab_actions.png b/doc/integration/gitlab_actions.png Binary files differnew file mode 100644 index 00000000000..b08f54d137b --- /dev/null +++ b/doc/integration/gitlab_actions.png diff --git a/doc/integration/gitlab_app.png b/doc/integration/gitlab_app.png Binary files differnew file mode 100644 index 00000000000..3f9391a821b --- /dev/null +++ b/doc/integration/gitlab_app.png diff --git a/doc/integration/gitlab_buttons_in_gmail.md b/doc/integration/gitlab_buttons_in_gmail.md new file mode 100644 index 00000000000..a9885cef109 --- /dev/null +++ b/doc/integration/gitlab_buttons_in_gmail.md @@ -0,0 +1,28 @@ +# GitLab buttons in Gmail + +GitLab supports [Google actions in email](https://developers.google.com/gmail/markup/actions/actions-overview). + +If correctly setup, emails that require an action will be marked in Gmail. + + + +To get this functioning, you need to be registered with Google. +[See how to register with google in this document.](https://developers.google.com/gmail/markup/registering-with-google) + +To aid the registering with google, GitLab offers a rake task that will send an email to google whitelisting email address from your GitLab server. + +To check what would be sent to the google email address, run the rake task: + +```bash +bundle exec rake gitlab:mail_google_schema_whitelisting RAILS_ENV=production +``` + +**This will not send the email but give you the output of how the mail will look.** + +Copy the output of the rake task to [google email markup tester](https://www.google.com/webmasters/markup-tester/u/0/) and press "Validate". + +If you receive "No errors detected" message from the tester you can send the email using: + +```bash +bundle exec rake gitlab:mail_google_schema_whitelisting RAILS_ENV=production SEND=true +``` diff --git a/doc/integration/google.md b/doc/integration/google.md index 7a78aff8ea4..e1c14c7c948 100644 --- a/doc/integration/google.md +++ b/doc/integration/google.md @@ -27,27 +27,50 @@ To enable the Google OAuth2 OmniAuth provider you must register your application - Authorized redirect URI: 'https://gitlab.example.com/users/auth/google_oauth2/callback' 1. Under the heading "Client ID for web application" you should see a Client ID and Client secret (see screenshot). Keep this page open as you continue configuration.  -1. On your GitLab server, open the configuration file. +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For instalations from source: ```sh - cd /home/git/gitlab + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml + sudo -u git -H editor config/gitlab.yml ``` -1. Find the section dealing with OmniAuth. See [Initial OmniAuth Configuration](README.md#initial-omniauth-configuration) for more details. +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "google_oauth2", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "args" => { "access_type" => "offline", "approval_prompt" => '' } + } + ] + ``` -1. Under `providers:` uncomment (or add) lines that look like the following: + For installations from source: ``` - - { name: 'google_oauth2', app_id: 'YOUR APP ID', - app_secret: 'YOUR APP SECRET', - args: { access_type: 'offline', approval_prompt: '' } } + - { name: 'google_oauth2', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { access_type: 'offline', approval_prompt: '' } } ``` -1. Change 'YOUR APP ID' to the client ID from the GitHub application page from step 7. +1. Change 'YOUR_APP_ID' to the client ID from the Google Developer page from step 10. -1. Change 'YOUR APP SECRET' to the client secret from the GitHub application page from step 7. +1. Change 'YOUR_APP_SECRET' to the client secret from the Google Developer page from step 10. 1. Save the configuration file. diff --git a/doc/integration/ldap.md b/doc/integration/ldap.md index ee472ac3e3b..125ce31b521 100644 --- a/doc/integration/ldap.md +++ b/doc/integration/ldap.md @@ -6,6 +6,98 @@ The first time a user signs in with LDAP credentials, GitLab will create a new G GitLab user attributes such as nickname and email will be copied from the LDAP user entry. +## Configuring GitLab for LDAP integration + +To enable GitLab LDAP integration you need to add your LDAP server settings in `/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml`. +In GitLab Enterprise Edition you can have multiple LDAP servers connected to one GitLab server. + +Please note that before version 7.4, GitLab used a different syntax for configuring LDAP integration. +The old LDAP integration syntax still works in GitLab 7.4. +If your `gitlab.rb` or `gitlab.yml` file contains LDAP settings in both the old syntax and the new syntax, only the __old__ syntax will be used by GitLab. + +```ruby +# For omnibus packages +gitlab_rails['ldap_enabled'] = true +gitlab_rails['ldap_servers'] = YAML.load <<-EOS # remember to close this block with 'EOS' below +main: # 'main' is the GitLab 'provider ID' of this LDAP server + ## label + # + # A human-friendly name for your LDAP server. It is OK to change the label later, + # for instance if you find out it is too large to fit on the web page. + # + # Example: 'Paris' or 'Acme, Ltd.' + label: 'LDAP' + + host: '_your_ldap_server' + port: 389 + uid: 'sAMAccountName' + method: 'plain' # "tls" or "ssl" or "plain" + bind_dn: '_the_full_dn_of_the_user_you_will_bind_with' + password: '_the_password_of_the_bind_user' + + # This setting specifies if LDAP server is Active Directory LDAP server. + # For non AD servers it skips the AD specific queries. + # If your LDAP server is not AD, set this to false. + active_directory: true + + # If allow_username_or_email_login is enabled, GitLab will ignore everything + # after the first '@' in the LDAP username submitted by the user on login. + # + # Example: + # - the user enters 'jane.doe@example.com' and 'p@ssw0rd' as LDAP credentials; + # - GitLab queries the LDAP server with 'jane.doe' and 'p@ssw0rd'. + # + # If you are using "uid: 'userPrincipalName'" on ActiveDirectory you need to + # disable this setting, because the userPrincipalName contains an '@'. + allow_username_or_email_login: false + + # Base where we can search for users + # + # Ex. ou=People,dc=gitlab,dc=example + # + base: '' + + # Filter LDAP users + # + # Format: RFC 4515 http://tools.ietf.org/search/rfc4515 + # Ex. (employeeType=developer) + # + # Note: GitLab does not support omniauth-ldap's custom filter syntax. + # + user_filter: '' + +# GitLab EE only: add more LDAP servers +# Choose an ID made of a-z and 0-9 . This ID will be stored in the database +# so that GitLab can remember which LDAP server a user belongs to. +# uswest2: +# label: +# host: +# .... +EOS +``` + +If you are getting 'Connection Refused' errors when trying to connect to the LDAP server please double-check the LDAP `port` and `method` settings used by GitLab. +Common combinations are `method: 'plain'` and `port: 389`, OR `method: 'ssl'` and `port: 636`. + +If you are using a GitLab installation from source you can find the LDAP settings in `/home/git/gitlab/config/gitlab.yml`: + +``` +production: + # snip... + ldap: + enabled: false + servers: + main: # 'main' is the GitLab 'provider ID' of this LDAP server + ## label + # + # A human-friendly name for your LDAP server. It is OK to change the label later, + # for instance if you find out it is too large to fit on the web page. + # + # Example: 'Paris' or 'Acme, Ltd.' + label: 'LDAP' + # snip... +``` + ## Enabling LDAP sign-in for existing GitLab users When a user signs in to GitLab with LDAP for the first time, and their LDAP email address is the primary email address of an existing GitLab user, then the LDAP DN will be associated with the existing user. @@ -24,15 +116,22 @@ If you want to limit all GitLab access to a subset of the LDAP users on your LDA The filter must comply with [RFC 4515](http://tools.ietf.org/search/rfc4515). ```ruby -# For omnibus-gitlab -gitlab_rails['ldap_user_filter'] = '(employeeType=developer)' +# For omnibus packages; new LDAP server syntax +gitlab_rails['ldap_servers'] = YAML.load <<-EOS +main: + # snip... + user_filter: '(employeeType=developer)' +EOS ``` ```yaml -# For installations from source +# For installations from source; new LDAP server syntax production: ldap: - user_filter: '(employeeType=developer)' + servers: + main: + # snip... + user_filter: '(employeeType=developer)' ``` Tip: if you want to limit access to the nested members of an Active Directory group you can use the following syntax: diff --git a/doc/integration/oauth_provider.md b/doc/integration/oauth_provider.md new file mode 100644 index 00000000000..192c321f712 --- /dev/null +++ b/doc/integration/oauth_provider.md @@ -0,0 +1,35 @@ +## GitLab as OAuth2 authentication service provider + +This document is about using GitLab as an OAuth authentication service provider to sign into other services. +If you want to use other OAuth authentication service providers to sign into GitLab please see the [OAuth2 client documentation](../api/oauth2.md) + +OAuth2 provides client applications a 'secure delegated access' to server resources on behalf of a resource owner. Or you can allow users to sign in to your application with their GitLab.com account. +In fact OAuth allows to issue access token to third-party clients by an authorization server, +with the approval of the resource owner, or end-user. +Mostly, OAuth2 is using for SSO (Single sign-on). But you can find a lot of different usages for this functionality. +For example, our feature 'GitLab Importer' is using OAuth protocol to give an access to repositories without sharing user credentials to GitLab.com account. +Also GitLab.com application can be used for authentication to your GitLab instance if needed [GitLab OmniAuth](gitlab.md). + +GitLab has two ways to add new OAuth2 application to an instance, you can add application as regular user and through admin area. So GitLab actually can have an instance-wide and a user-wide applications. There is no defferences between them except the different permission levels. + +### Adding application through profile +Go to your profile section 'Application' and press button 'New Application' + + + +After this you will see application form, where "Name" is arbitrary name, "Redirect URI" is URL in your app where users will be sent after authorization on GitLab.com. + + + +### Authorized application +Every application you authorized will be shown in your "Authorized application" sections. + + + +At any time you can revoke access just clicking button "Revoke" + +### OAuth applications in admin area + +If you want to create application that does not belong to certain user you can create it from admin area + +
\ No newline at end of file diff --git a/doc/integration/oauth_provider/admin_application.png b/doc/integration/oauth_provider/admin_application.png Binary files differnew file mode 100644 index 00000000000..a5f34512aa8 --- /dev/null +++ b/doc/integration/oauth_provider/admin_application.png diff --git a/doc/integration/oauth_provider/application_form.png b/doc/integration/oauth_provider/application_form.png Binary files differnew file mode 100644 index 00000000000..ae135db2627 --- /dev/null +++ b/doc/integration/oauth_provider/application_form.png diff --git a/doc/integration/oauth_provider/authorized_application.png b/doc/integration/oauth_provider/authorized_application.png Binary files differnew file mode 100644 index 00000000000..d3ce05be9cc --- /dev/null +++ b/doc/integration/oauth_provider/authorized_application.png diff --git a/doc/integration/oauth_provider/user_wide_applications.png b/doc/integration/oauth_provider/user_wide_applications.png Binary files differnew file mode 100644 index 00000000000..719e1974068 --- /dev/null +++ b/doc/integration/oauth_provider/user_wide_applications.png diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index 00adae58dfa..24f7b4bb4b4 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -1,18 +1,47 @@ # OmniAuth -GitLab leverages OmniAuth to allow users to sign in using Twitter, GitHub, and other popular services. Configuring +GitLab leverages OmniAuth to allow users to sign in using Twitter, GitHub, and other popular services. -OmniAuth does not prevent standard GitLab authentication or LDAP (if configured) from continuing to work. Users can choose to sign in using any of the configured mechanisms. +Configuring OmniAuth does not prevent standard GitLab authentication or LDAP (if configured) from continuing to work. Users can choose to sign in using any of the configured mechanisms. - [Initial OmniAuth Configuration](#initial-omniauth-configuration) - [Supported Providers](#supported-providers) - [Enable OmniAuth for an Existing User](#enable-omniauth-for-an-existing-user) +- [OmniAuth configuration sample when using Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master#omniauth-google-twitter-github-login) ## Initial OmniAuth Configuration -Before configuring individual OmniAuth providers there are a few global settings that need to be verified. +Before configuring individual OmniAuth providers there are a few global settings that are in common for all providers that we need to consider. -1. Open the configuration file. +- Omniauth needs to be enabled, see details below for example. +- `allow_single_sign_on` defaults to `false`. If `false` users must be created manually or they will not be able to +sign in via OmniAuth. +- `block_auto_created_users` defaults to `true`. If `true` auto created users will be blocked by default and will +have to be unblocked by an administrator before they are able to sign in. +- **Note:** If you set `allow_single_sign_on` to `true` and `block_auto_created_users` to `false` please be aware +that any user on the Internet will be able to successfully sign in to your GitLab without administrative approval. + +If you want to change these settings: + +* **For omnibus package** + + Open the configuration file: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + and change + + ``` + gitlab_rails['omniauth_enabled'] = true + gitlab_rails['omniauth_allow_single_sign_on'] = false + gitlab_rails['block_auto_created_users'] = true + ``` + +* **For installations from source** + + Open the configuration file: ```sh cd /home/git/gitlab @@ -20,13 +49,13 @@ Before configuring individual OmniAuth providers there are a few global settings sudo -u git -H editor config/gitlab.yml ``` -1. Find the section dealing with OmniAuth. The section will look similar to the following. + and change the following section ``` - ## OmniAuth settings + ## OmniAuth settings omniauth: # Allow login via Twitter, Google, etc. using OmniAuth providers - enabled: false + enabled: true # CAUTION! # This allows users to login without having a user account first (default: false). @@ -34,47 +63,15 @@ Before configuring individual OmniAuth providers there are a few global settings allow_single_sign_on: false # Locks down those users until they have been cleared by the admin (default: true). block_auto_created_users: true - - ## Auth providers - # Uncomment the following lines and fill in the data of the auth provider you want to use - # If your favorite auth provider is not listed you can use others: - # see https://github.com/gitlabhq/gitlab-public-wiki/wiki/Custom-omniauth-provider-configurations - # The 'app_id' and 'app_secret' parameters are always passed as the first two - # arguments, followed by optional 'args' which can be either a hash or an array. - providers: - # - { name: 'google_oauth2', app_id: 'YOUR APP ID', - # app_secret: 'YOUR APP SECRET', - # args: { access_type: 'offline', approval_prompt: '' } } - # - { name: 'twitter', app_id: 'YOUR APP ID', - # app_secret: 'YOUR APP SECRET'} - # - { name: 'github', app_id: 'YOUR APP ID', - # app_secret: 'YOUR APP SECRET', - # args: { scope: 'user:email' } } - # - {"name": 'shibboleth', - # args: { shib_session_id_field: "HTTP_SHIB_SESSION_ID", - # shib_application_id_field: "HTTP_SHIB_APPLICATION_ID", - # uid_field: "HTTP_EPPN", - # name_field: "HTTP_CN", - # info_fields: {"email": "HTTP_MAIL" } } } - ``` -1. Change `enabled` to `true`. - -1. Consider the next two configuration options: `allow_single_sign_on` and `block_auto_created_users`. - - - `allow_single_sign_on` defaults to `false`. If `false` users must be created manually or they will not be able to - sign in via OmniAuth. - - `block_auto_created_users` defaults to `true`. If `true` auto created users will be blocked by default and will - have to be unblocked by an administrator before they are able to sign in. - - **Note:** If you set `allow_single_sign_on` to `true` and `block_auto_created_users` to `false` please be aware - that any user on the Internet will be able to successfully sign in to your GitLab without administrative approval. - -1. Choose one or more of the Supported Providers below to continue configuration. +Now we can choose one or more of the Supported Providers below to continue configuration. ## Supported Providers - [GitHub](github.md) +- [Bitbucket](bitbucket.md) +- [GitLab.com](gitlab.md) - [Google](google.md) - [Shibboleth](shibboleth.md) - [Twitter](twitter.md) diff --git a/doc/integration/redmine_configuration.png b/doc/integration/redmine_configuration.png Binary files differnew file mode 100644 index 00000000000..6b145363229 --- /dev/null +++ b/doc/integration/redmine_configuration.png diff --git a/doc/integration/redmine_service_template.png b/doc/integration/redmine_service_template.png Binary files differnew file mode 100644 index 00000000000..1159eb5b964 --- /dev/null +++ b/doc/integration/redmine_service_template.png diff --git a/doc/integration/shibboleth.md b/doc/integration/shibboleth.md index 78317a5c0f2..6258e5f1030 100644 --- a/doc/integration/shibboleth.md +++ b/doc/integration/shibboleth.md @@ -2,19 +2,19 @@ This documentation is for enabling shibboleth with gitlab-omnibus package. -In order to enable Shibboleth support in gitlab we need to use Apache instead of Nginx (It may be possible to use Nginx, however I did not found way to easily configure nginx that is bundled in gitlab-omnibus package). Apache uses mod_shib2 module for shibboleth authentication and can pass attributes as headers to omniauth-shibboleth provider. +In order to enable Shibboleth support in gitlab we need to use Apache instead of Nginx (It may be possible to use Nginx, however I did not found way to easily configure Nginx that is bundled in gitlab-omnibus package). Apache uses mod_shib2 module for shibboleth authentication and can pass attributes as headers to omniauth-shibboleth provider. To enable the Shibboleth OmniAuth provider you must: -1. Configure Apache shibboleth module. Installation and configuration of module it self is out of scope of this document. +1. Configure Apache shibboleth module. Installation and configuration of module it self is out of scope of this document. Check https://wiki.shibboleth.net/ for more info. -1. You can find Apache config in gitlab-reciepes (https://github.com/gitlabhq/gitlab-recipes/blob/master/web-server/apache/gitlab-ssl.conf) +1. You can find Apache config in gitlab-recipes (https://github.com/gitlabhq/gitlab-recipes/blob/master/web-server/apache/gitlab-ssl.conf) Following changes are needed to enable shibboleth: -protect omniauth-shibboleth callback url: +protect omniauth-shibboleth callback URL: ``` <Location /users/auth/shibboleth/callback> AuthType shibboleth @@ -32,25 +32,25 @@ protect omniauth-shibboleth callback url: SetHandler shib </Location> ``` -exclude shibboleth urls from rewriting, add "RewriteCond %{REQUEST_URI} !/Shibboleth.sso" and "RewriteCond %{REQUEST_URI} !/shibboleth-sp", config should look like this: +exclude shibboleth URLs from rewriting, add "RewriteCond %{REQUEST_URI} !/Shibboleth.sso" and "RewriteCond %{REQUEST_URI} !/shibboleth-sp", config should look like this: ``` - #apache equivalent of nginx try files + # Apache equivalent of Nginx try files RewriteEngine on RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f - RewriteCond %{REQUEST_URI} !/Shibboleth.sso - RewriteCond %{REQUEST_URI} !/shibboleth-sp + RewriteCond %{REQUEST_URI} !/Shibboleth.sso + RewriteCond %{REQUEST_URI} !/shibboleth-sp RewriteRule .* http://127.0.0.1:8080%{REQUEST_URI} [P,QSA] RequestHeader set X_FORWARDED_PROTO 'https' ``` -1. Edit /etc/gitlab/gitlab.rb configuration file, your shibboleth attributes should be in form of "HTTP_ATTRIBUTE" and you should addjust them to your need and environment. Add any other configuration you need. +1. Edit /etc/gitlab/gitlab.rb configuration file, your shibboleth attributes should be in form of "HTTP_ATTRIBUTE" and you should addjust them to your need and environment. Add any other configuration you need. -File it should look like this: +File should look like this: ``` external_url 'https://gitlab.example.com' gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' -# disable nginx +# disable Nginx nginx['enable'] = false gitlab_rails['omniauth_allow_single_sign_on'] = true @@ -70,7 +70,7 @@ gitlab_rails['omniauth_providers'] = [ ] ``` -1. Save changes and reconfigure gitlab: +1. Save changes and reconfigure gitlab: ``` sudo gitlab-ctl reconfigure ``` diff --git a/doc/integration/slack.md b/doc/integration/slack.md index 95cb0c6fae2..2fd22c513ad 100644 --- a/doc/integration/slack.md +++ b/doc/integration/slack.md @@ -4,15 +4,23 @@ To enable Slack integration you must create an Incoming WebHooks integration on Slack; -1. Sign in to [Slack](https://slack.com) (https://YOURSUBDOMAIN.slack.com/services) -1. Click on the Integrations menu at the top of the page. -1. Add a new Integration. +1. [Sign in to Slack](https://slack.com/signin) + +1. Select **Configure Integrations** from the dropdown next to your team name. + +1. Select the **All Services** tab + +1. Click **Add** next to Incoming Webhooks + 1. Pick Incoming WebHooks -1. Choose the channel name you want to send notifications to, in the Settings section -1. Add Integrations. - - Optional step; You can change bot's name and avatar by clicking "change the name of your bot", and "change the icon" after that you have to click "Save settings". -Now, Slack is ready to get external hooks. Before you leave this page don't forget to get the Token that you'll need on GitLab. You can find it by clicking Expand button, located in the "Instructions for creating Incoming WebHooks" section. It's a random alpha-numeric text 24 characters long. +1. Choose the channel name you want to send notifications to + +1. Click **Add Incoming WebHooks Integration**Add Integrations. + - Optional step; You can change bot's name and avatar by clicking modifying the bot name or avatar under **Integration Settings**. + +1. Copy the **Webhook URL**, we'll need this later for GitLab. + ## On GitLab @@ -26,10 +34,8 @@ After Slack is ready we need to setup GitLab. Here are the steps to achieve this 1. Fill in your Slack details - - Mark as active it - - Type your subdomain's prefix (If your subdomain is https://somedomain.slack.com you only have to type the somedomain) - - Type in the token you got from Slack - - Type in the channel name you want to use (eg. #announcements) + - Mark it as active + - Paste in the webhook URL you got from Slack Have fun :) diff --git a/doc/integration/twitter.md b/doc/integration/twitter.md index d1b52927d30..fe9091ad9a8 100644 --- a/doc/integration/twitter.md +++ b/doc/integration/twitter.md @@ -13,7 +13,7 @@ To enable the Twitter OmniAuth provider you must register your application with something else descriptive. - Description: Create a description. - Website: The URL to your GitLab installation. 'https://gitlab.example.com' - - Callback URL: 'https://gitlab.example.com/users/auth/github/callback' + - Callback URL: 'https://gitlab.example.com/users/auth/twitter/callback' - Agree to the "Rules of the Road."  @@ -33,25 +33,46 @@ To enable the Twitter OmniAuth provider you must register your application with 1. On your GitLab server, open the configuration file. + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For instalations from source: + ```sh - cd /home/git/gitlab + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml + sudo -u git -H editor config/gitlab.yml ``` -1. Find the section dealing with OmniAuth. See [Initial OmniAuth Configuration](README.md#initial-omniauth-configuration) -for more details. +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "twitter", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET" + } + ] + ``` -1. Under `providers:` uncomment (or add) lines that look like the following: + For installations from source: ``` - - { name: 'twitter', app_id: 'YOUR APP ID', - app_secret: 'YOUR APP SECRET' } + - { name: 'twitter', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET' } ``` -1. Change 'YOUR APP ID' to the API key from Twitter page in step 11. +1. Change 'YOUR_APP_ID' to the API key from Twitter page in step 11. -1. Change 'YOUR APP SECRET' to the API secret from the Twitter page in step 11. +1. Change 'YOUR_APP_SECRET' to the API secret from the Twitter page in step 11. 1. Save the configuration file. diff --git a/doc/logs/logs.md b/doc/logs/logs.md new file mode 100644 index 00000000000..ec0109a426f --- /dev/null +++ b/doc/logs/logs.md @@ -0,0 +1,102 @@ +## Log system +GitLab has advanced log system so everything is logging and you can analize your instance using various system log files. +In addition to system log files, GitLab Enterprise Edition comes with Audit Events. Find more about them [in Audit Events documentation](http://doc.gitlab.com/ee/administration/audit_events.html) + +System log files are typically plain text in a standard log file format. This guide talks about how to read and use these system log files. + +#### production.log +This file lives in `/var/log/gitlab/gitlab-rails/production.log` for omnibus package or in `/home/git/gitlab/logs/production.log` for installations from the source. + +This file contains information about all performed requests. You can see url and type of request, IP address and what exactly parts of code were involved to service this particular request. Also you can see all SQL request that have been performed and how much time it took. +This task is more useful for GitLab contributors and developers. Use part of this log file when you are going to report bug. + +``` +Started GET "/gitlabhq/yaml_db/tree/master" for 168.111.56.1 at 2015-02-12 19:34:53 +0200 +Processing by Projects::TreeController#show as HTML + Parameters: {"project_id"=>"gitlabhq/yaml_db", "id"=>"master"} + + ... [CUT OUT] + + amespaces"."created_at" DESC, "namespaces"."id" DESC LIMIT 1[0m [["id", 26]] + [1m[35mCACHE (0.0ms)[0m SELECT "members".* FROM "members" WHERE "members"."source_type" = 'Project' AND "members"."type" IN ('ProjectMember') AND "members"."source_id" = $1 AND "members"."source_type" = $2 AND "members"."user_id" = 1 ORDER BY "members"."created_at" DESC, "members"."id" DESC LIMIT 1 [["source_id", 18], ["source_type", "Project"]] + [1m[36mCACHE (0.0ms)[0m [1mSELECT "members".* FROM "members" WHERE "members"."source_type" = 'Project' AND "members". + [1m[36m (1.4ms)[0m [1mSELECT COUNT(*) FROM "merge_requests" WHERE "merge_requests"."target_project_id" = $1 AND ("merge_requests"."state" IN ('opened','reopened'))[0m [["target_project_id", 18]] + Rendered layouts/nav/_project.html.haml (28.0ms) + Rendered layouts/_collapse_button.html.haml (0.2ms) + Rendered layouts/_flash.html.haml (0.1ms) + Rendered layouts/_page.html.haml (32.9ms) +Completed 200 OK in 166ms (Views: 117.4ms | ActiveRecord: 27.2ms) +``` +In this example we can see that server processed HTTP request with url `/gitlabhq/yaml_db/tree/master` from IP 168.111.56.1 at 2015-02-12 19:34:53 +0200. Also we can see that request was processed by Projects::TreeController. + +#### application.log +This file lives in `/var/log/gitlab/gitlab-rails/application.log` for omnibus package or in `/home/git/gitlab/logs/application.log` for installations from the source. + +This log file helps you discover events happening in your instance such as user creation, project removing and so on. + +``` +October 06, 2014 11:56: User "Administrator" (admin@example.com) was created +October 06, 2014 11:56: Documentcloud created a new project "Documentcloud / Underscore" +October 06, 2014 11:56: Gitlab Org created a new project "Gitlab Org / Gitlab Ce" +October 07, 2014 11:25: User "Claudie Hodkiewicz" (nasir_stehr@olson.co.uk) was removed +October 07, 2014 11:25: Project "project133" was removed +``` +#### githost.log +This file lives in `/var/log/gitlab/gitlab-rails/githost.log` for omnibus package or in `/home/git/gitlab/logs/githost.log` for installations from the source. + +The GitLab has to interact with git repositories but in some rare cases something can go wrong and in this case you will know what exactly happened. This log file contains all failed requests from GitLab to git repository. In majority of cases this file will be useful for developers only. +``` +December 03, 2014 13:20 -> ERROR -> Command failed [1]: /usr/bin/git --git-dir=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq/.git --work-tree=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq merge --no-ff -mMerge branch 'feature_conflict' into 'feature' source/feature_conflict + +error: failed to push some refs to '/Users/vsizov/gitlab-development-kit/repositories/gitlabhq/gitlab_git.git' +``` + +#### satellites.log +This file lives in `/var/log/gitlab/gitlab-rails/satellites.log` for omnibus package or in `/home/git/gitlab/logs/satellites.log` for installations from the source. + +In some cases GitLab should perform write actions to git repository, for example when it is needed to merge the merge request or edit a file with online editor. If something went wrong you can look into this file to find out what exactly happened. +``` +October 07, 2014 11:36: Failed to create satellite for Chesley Weimann III / project1817 +October 07, 2014 11:36: PID: 1872: git clone /Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/repositories/conrad6841/gitlabhq.git /Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/conrad6841/gitlabhq +October 07, 2014 11:36: PID: 1872: -> fatal: repository '/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/repositories/conrad6841/gitlabhq.git' does not exist +``` + +#### sidekiq.log +This file lives in `/var/log/gitlab/gitlab-rails/sidekiq.log` for omnibus package or in `/home/git/gitlab/logs/sidekiq.log` for installations from the source. + +GitLab uses background jobs for processing tasks which can take a long time. All information about processing these jobs are writing down to this file. +``` +2014-06-10T07:55:20Z 2037 TID-tm504 ERROR: /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/1.9.1/gems/redis-3.0.7/lib/redis/client.rb:228:in `read' +2014-06-10T18:18:26Z 14299 TID-55uqo INFO: Booting Sidekiq 3.0.0 with redis options {:url=>"redis://localhost:6379/0", :namespace=>"sidekiq"} +``` + +#### gitlab-shell.log +This file lives in `/var/log/gitlab/gitlab-shell/gitlab-shell.log` for omnibus package or in `/home/git/gitlab-shell/logs/sidekiq.log` for installations from the source. + +gitlab-shell is using by Gitlab for executing git commands and provide ssh access to git repositories. + +``` +I, [2015-02-13T06:17:00.671315 #9291] INFO -- : Adding project root/example.git at </var/opt/gitlab/git-data/repositories/root/dcdcdcdcd.git>. +I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory and simlinking global hooks directory for /var/opt/gitlab/git-data/repositories/root/example.git. +``` + +#### unicorn_stderr.log +This file lives in `/var/log/gitlab/unicorn/unicorn_stderr.log` for omnibus package or in `/home/git/gitlab/logs/unicorn_stderr.log` for installations from the source. + +Unicorn is a high-performance forking Web server which is used for serving GitLab application. You can look at this log, for example, if your application does not respond. This log cantains all information about state of unicorn processes at any given time. + +``` +I, [2015-02-13T06:14:46.680381 #9047] INFO -- : Refreshing Gem list +I, [2015-02-13T06:14:56.931002 #9047] INFO -- : listening on addr=127.0.0.1:8080 fd=12 +I, [2015-02-13T06:14:56.931381 #9047] INFO -- : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=13 +I, [2015-02-13T06:14:56.936638 #9047] INFO -- : master process ready +I, [2015-02-13T06:14:56.946504 #9092] INFO -- : worker=0 spawned pid=9092 +I, [2015-02-13T06:14:56.946943 #9092] INFO -- : worker=0 ready +I, [2015-02-13T06:14:56.947892 #9094] INFO -- : worker=1 spawned pid=9094 +I, [2015-02-13T06:14:56.948181 #9094] INFO -- : worker=1 ready +W, [2015-02-13T07:16:01.312916 #9094] WARN -- : #<Unicorn::HttpServer:0x0000000208f618>: worker (pid: 9094) exceeds memory limit (320626688 bytes > 247066940 bytes) +W, [2015-02-13T07:16:01.313000 #9094] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 9094) alive: 3621 sec (trial 1) +I, [2015-02-13T07:16:01.530733 #9047] INFO -- : reaped #<Process::Status: pid 9094 exit 0> worker=1 +I, [2015-02-13T07:16:01.534501 #13379] INFO -- : worker=1 spawned pid=13379 +I, [2015-02-13T07:16:01.534848 #13379] INFO -- : worker=1 ready +``` diff --git a/doc/markdown/markdown.md b/doc/markdown/markdown.md index 0f63eca1f61..ddf1bbc6ee4 100644 --- a/doc/markdown/markdown.md +++ b/doc/markdown/markdown.md @@ -6,7 +6,7 @@ * [Newlines](#newlines) * [Multiple underscores in words](#multiple-underscores-in-words) -* [URL autolinking](#url-autolinking) +* [URL auto-linking](#url-autolinking) * [Code and Syntax Highlighting](#code-and-syntax-highlighting) * [Emoji](#emoji) * [Special GitLab references](#special-gitlab-references) @@ -40,7 +40,7 @@ You can use GFM in - milestones - wiki pages -You can also use other rich text files in GitLab. You might have to install a depency to do so. Please see the [github-markup gem readme](https://github.com/gitlabhq/markup#markups) for more information. +You can also use other rich text files in GitLab. You might have to install a dependency to do so. Please see the [github-markup gem readme](https://github.com/gitlabhq/markup#markups) for more information. ## Newlines @@ -68,7 +68,7 @@ It is not reasonable to italicize just _part_ of a word, especially when you're perform_complicated_task do_this_and_do_that_and_another_thing -## URL autolinking +## URL auto-linking GFM will autolink standard URLs you copy and paste into your text. So if you want to link to a URL (instead of a textural link), you can simply put the URL in verbatim and it will be turned into a link to that URL. @@ -140,25 +140,25 @@ But let's throw in a <b>tag</b>. ## Emoji - Sometimes you want to be a :ninja: and add some :glowing_star: to your :speech_balloon:. Well we have a gift for you: + Sometimes you want to :monkey: around a bit and add some :star2: to your :speech_balloon:. Well we have a gift for you: - :high_voltage_sign: You can use emoji anywhere GFM is supported. :victory_hand: + :zap: You can use emoji anywhere GFM is supported. :v: - You can use it to point out a :bug: or warn about :speak_no_evil_monkey: patches. And if someone improves your really :snail: code, send them some :cake:. People will :heart: you for that. + You can use it to point out a :bug: or warn about :speak_no_evil: patches. And if someone improves your really :snail: code, send them some :birthday:. People will :heart: you for that. - If you are new to this, don't be :fearful_face:. You can easily join the emoji :family:. All you need to do is to look up on the supported codes. + If you are new to this, don't be :fearful:. You can easily join the emoji :family:. All you need to do is to look up on the supported codes. - Consult the [Emoji Cheat Sheet](https://www.dropbox.com/s/b9xaqb977s6d8w1/cheat_sheet.pdf) for a list of all supported emoji codes. :thumbsup: + Consult the [Emoji Cheat Sheet](http://emoji.codes) for a list of all supported emoji codes. :thumbsup: -Sometimes you want to be a :ninja: and add some :glowing_star: to your :speech_balloon:. Well we have a gift for you: +Sometimes you want to :monkey: around a bit and add some :star2: to your :speech_balloon:. Well we have a gift for you: -:high_voltage_sign: You can use emoji anywhere GFM is supported. :victory_hand: +:zap: You can use emoji anywhere GFM is supported. :v: -You can use it to point out a :bug: or warn about :speak_no_evil_monkey: patches. And if someone improves your really :snail: code, send them some :cake:. People will :heart: you for that. +You can use it to point out a :bug: or warn about :speak_no_evil: patches. And if someone improves your really :snail: code, send them some :birthday:. People will :heart: you for that. -If you are new to this, don't be :fearful_face:. You can easily join the emoji :family:. All you need to do is to look up on the supported codes. +If you are new to this, don't be :fearful:. You can easily join the emoji :family:. All you need to do is to look up on the supported codes. -Consult the [Emoji Cheat Sheet](https://www.dropbox.com/s/b9xaqb977s6d8w1/cheat_sheet.pdf) for a list of all supported emoji codes. :thumbsup: +Consult the [Emoji Cheat Sheet](http://emoji.codes) for a list of all supported emoji codes. :thumbsup: ## Special GitLab References @@ -170,7 +170,7 @@ GFM will turn that reference into a link so you can navigate between them easily GFM will recognize the following: -- @foo : for team members +- @foo : for specific team members or groups - @all : for the whole team - #123 : for issues - !123 : for merge requests @@ -250,17 +250,17 @@ The IDs are generated from the content of the header according to the following For example: ``` -###### ..Ab_c-d. e [anchor](url) .. +###### ..Ab_c-d. e [anchor](URL) .. ``` which renders as: -###### ..Ab_c-d. e [anchor](url) .. +###### ..Ab_c-d. e [anchor](URL) .. will first be converted by step 1) into a string like: ``` -..Ab_c-d. e <a href="url">anchor</a> <img src="url" alt="alt text"/>.. +..Ab_c-d. e <a href="URL">anchor</a> <img src="URL" alt="alt text"/>.. ``` After removing the tags in step 2) we get: @@ -277,8 +277,8 @@ ab_c-d-e-anchor Note in particular how: -- for markdown anchors `[text](url)`, only the `text` is used -- markdown images `` are completely ignored +- for markdown anchors `[text](URL)`, only the `text` is used +- markdown images `` are completely ignored ## Emphasis @@ -420,6 +420,8 @@ Quote break. You can also use raw HTML in your Markdown, and it'll mostly work pretty well. +Note that inline HTML is disabled in the default Gitlab configuration, although it is [possible](https://github.com/gitlabhq/gitlabhq/pull/8007/commits) for the system administrator to enable it. + ```no-highlight <dl> <dt>Definition list</dt> @@ -542,6 +544,10 @@ This line is separated from the one above by two newlines, so it will be a *sepa This line is also a separate paragraph, but... This line is only separated by a single newline, so it's a separate line in the *same paragraph*. + +This line is also a separate paragraph, and... +This line is on its own line, because the previous line ends with two +spaces. ``` Here's a line for us to start with. @@ -551,6 +557,10 @@ This line is separated from the one above by two newlines, so it will be a *sepa This line is also begins a separate paragraph, but... This line is only separated by a single newline, so it's a separate line in the *same paragraph*. +This line is also a separate paragraph, and... +This line is on its own line, because the previous line ends with two +spaces. + ## Tables Tables aren't part of the core Markdown spec, but they are part of GFM and Markdown Here supports them. @@ -569,6 +579,10 @@ Code above produces next output: | cell 1 | cell 2 | | cell 3 | cell 4 | +**Note** + +The row of dashes between the table header and body must have at least three dashes in each column. + ## References - This document leveraged heavily from the [Markdown-Cheatsheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet). diff --git a/doc/operations/README.md b/doc/operations/README.md new file mode 100644 index 00000000000..f1456c6c8e2 --- /dev/null +++ b/doc/operations/README.md @@ -0,0 +1,4 @@ +# GitLab operations + +- [Sidekiq MemoryKiller](sidekiq_memory_killer.md) +- [Cleaning up Redis sessions](cleaning_up_redis_sessions.md) diff --git a/doc/operations/cleaning_up_redis_sessions.md b/doc/operations/cleaning_up_redis_sessions.md new file mode 100644 index 00000000000..93521e976d5 --- /dev/null +++ b/doc/operations/cleaning_up_redis_sessions.md @@ -0,0 +1,52 @@ +# Cleaning up stale Redis sessions + +Since version 6.2, GitLab stores web user sessions as key-value pairs in Redis. +Prior to GitLab 7.3, user sessions did not automatically expire from Redis. If +you have been running a large GitLab server (thousands of users) since before +GitLab 7.3 we recommend cleaning up stale sessions to compact the Redis +database after you upgrade to GitLab 7.3. You can also perform a cleanup while +still running GitLab 7.2 or older, but in that case new stale sessions will +start building up again after you clean up. + +In GitLab versions prior to 7.3.0, the session keys in Redis are 16-byte +hexadecimal values such as '976aa289e2189b17d7ef525a6702ace9'. Starting with +GitLab 7.3.0, the keys are +prefixed with 'session:gitlab:', so they would look like +'session:gitlab:976aa289e2189b17d7ef525a6702ace9'. Below we describe how to +remove the keys in the old format. + +First we define a shell function with the proper Redis connection details. + +``` +rcli() { + # This example works for Omnibus installations of GitLab 7.3 or newer. For an + # installation from source you will have to change the socket path and the + # path to redis-cli. + sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@" +} + +# test the new shell function; the response should be PONG +rcli ping +``` + +Now we do a search to see if there are any session keys in the old format for +us to clean up. + +``` +# returns the number of old-format session keys in Redis +rcli keys '*' | grep '^[a-f0-9]\{32\}$' | wc -l +``` + +If the number is larger than zero, you can proceed to expire the keys from +Redis. If the number is zero there is nothing to clean up. + +``` +# Tell Redis to expire each matched key after 600 seconds. +rcli keys '*' | grep '^[a-f0-9]\{32\}$' | awk '{ print "expire", $0, 600 }' | rcli +# This will print '(integer) 1' for each key that gets expired. +``` + +Over the next 15 minutes (10 minutes expiry time plus 5 minutes Redis +background save interval) your Redis database will be compacted. If you are +still using GitLab 7.2, users who are not clicking around in GitLab during the +10 minute expiry window will be signed out of GitLab. diff --git a/doc/operations/sidekiq_memory_killer.md b/doc/operations/sidekiq_memory_killer.md new file mode 100644 index 00000000000..867b01b0d5a --- /dev/null +++ b/doc/operations/sidekiq_memory_killer.md @@ -0,0 +1,38 @@ +# Sidekiq MemoryKiller + +The GitLab Rails application code suffers from memory leaks. For web requests +this problem is made manageable using +[unicorn-worker-killer](https://github.com/kzk/unicorn-worker-killer) which +restarts Unicorn worker processes in between requests when needed. The Sidekiq +MemoryKiller applies the same approach to the Sidekiq processes used by GitLab +to process background jobs. + +Unlike unicorn-worker-killer, which is enabled by default for all GitLab +installations since GitLab 6.4, the Sidekiq MemoryKiller is enabled by default +_only_ for Omnibus packages. The reason for this is that the MemoryKiller +relies on Runit to restart Sidekiq after a memory-induced shutdown and GitLab +installations from source do not all use Runit or an equivalent. + +With the default settings, the MemoryKiller will cause a Sidekiq restart no +more often than once every 15 minutes, with the restart causing about one +minute of delay for incoming background jobs. + +## Configuring the MemoryKiller + +The MemoryKiller is controlled using environment variables. + +- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is + greater than 0, then after each Sidekiq job, the MemoryKiller will check the + RSS of the Sidekiq process that executed the job. If the RSS of the Sidekiq + process (expressed in kilobytes) exceeds SIDEKIQ_MEMORY_KILLER_MAX_RSS, a + delayed shutdown is triggered. The default value for Omnibus packages is set + [in the omnibus-gitlab + repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb). +- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults 900 seconds (15 minutes). When + a shutdown is triggered, the Sidekiq process will keep working normally for + another 15 minutes. +- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. When the grace + time has expired, the MemoryKiller tells Sidekiq to stop accepting new jobs. + Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells + Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must + restart Sidekiq. diff --git a/doc/permissions/permissions.md b/doc/permissions/permissions.md index d561868c8bb..c9928e11b2e 100644 --- a/doc/permissions/permissions.md +++ b/doc/permissions/permissions.md @@ -8,7 +8,6 @@ If a user is a GitLab administrator they receive all permissions. ## Project - | Action | Guest | Reporter | Developer | Master | Owner | |---------------------------------------|---------|------------|-------------|----------|--------| | Create new issue | ✓ | ✓ | ✓ | ✓ | ✓ | @@ -19,6 +18,7 @@ If a user is a GitLab administrator they receive all permissions. | Create new merge request | | | ✓ | ✓ | ✓ | | Create new branches | | | ✓ | ✓ | ✓ | | Push to non-protected branches | | | ✓ | ✓ | ✓ | +| Force push to non-protected branches | | | ✓ | ✓ | ✓ | | Remove non-protected branches | | | ✓ | ✓ | ✓ | | Add tags | | | ✓ | ✓ | ✓ | | Write a wiki | | | ✓ | ✓ | ✓ | @@ -28,6 +28,7 @@ If a user is a GitLab administrator they receive all permissions. | Add new team members | | | | ✓ | ✓ | | Push to protected branches | | | | ✓ | ✓ | | Enable/disable branch protection | | | | ✓ | ✓ | +| Turn on/off prot. branch push for devs| | | | ✓ | ✓ | | Rewrite/remove git tags | | | | ✓ | ✓ | | Edit project | | | | ✓ | ✓ | | Add deploy keys to project | | | | ✓ | ✓ | @@ -35,6 +36,8 @@ If a user is a GitLab administrator they receive all permissions. | Switch visibility level | | | | | ✓ | | Transfer project to another namespace | | | | | ✓ | | Remove project | | | | | ✓ | +| Force push to protected branches | | | | | | +| Remove protected branches | | | | | | ## Group @@ -46,4 +49,4 @@ If a user is a GitLab administrator they receive all permissions. | Manage group members | | | | | ✓ | | Remove group | | | | | ✓ | -Any user can remove himself from a group, unless he is the last Owner of the group. +Any user can remove themselves from a group, unless they are the last Owner of the group. diff --git a/doc/project_services/bamboo.md b/doc/project_services/bamboo.md new file mode 100644 index 00000000000..51668128c62 --- /dev/null +++ b/doc/project_services/bamboo.md @@ -0,0 +1,60 @@ +# Atlassian Bamboo CI Service + +GitLab provides integration with Atlassian Bamboo for continuous integration. +When configured, pushes to a project will trigger a build in Bamboo automatically. +Merge requests will also display CI status showing whether the build is pending, +failed, or completed successfully. It also provides a link to the Bamboo build +page for more information. + +Bamboo doesn't quite provide the same features as a traditional build system when +it comes to accepting webhooks and commit data. There are a few things that +need to be configured in a Bamboo build plan before GitLab can integrate. + +## Setup + +### Complete these steps in Bamboo: + +1. Navigate to a Bamboo build plan and choose 'Configure plan' from the 'Actions' +dropdown. +1. Select the 'Triggers' tab. +1. Click 'Add trigger'. +1. Enter a description such as 'GitLab trigger' +1. Choose 'Repository triggers the build when changes are committed' +1. Check one or more repositories checkboxes +1. Enter the GitLab IP address in the 'Trigger IP addresses' box. This is a +whitelist of IP addresses that are allowed to trigger Bamboo builds. +1. Save the trigger. +1. In the left pane, select a build stage. If you have multiple build stages +you want to select the last stage that contains the git checkout task. +1. Select the 'Miscellaneous' tab. +1. Under 'Pattern Match Labelling' put '${bamboo.repository.revision.number}' +in the 'Labels' box. +1. Save + +Bamboo is now ready to accept triggers from GitLab. Next, set up the Bamboo +service in GitLab + +### Complete these steps in GitLab: + +1. Navigate to the project you want to configure to trigger builds. +1. Select 'Settings' in the top navigation. +1. Select 'Services' in the left navigation. +1. Click 'Atlassian Bamboo CI' +1. Select the 'Active' checkbox. +1. Enter the base URL of your Bamboo server. 'https://bamboo.example.com' +1. Enter the build key from your Bamboo build plan. Build keys are a short, +all capital letter, identifier that is unique. It will be something like PR-BLD +1. If necessary, enter username and password for a Bamboo user that has +access to trigger the build plan. Leave these fields blank if you do not require +authentication. +1. Save or optionally click 'Test Settings'. Please note that 'Test Settings' +will actually trigger a build in Bamboo. + +## Troubleshooting + +If builds are not triggered, these are a couple of things to keep in mind. + +1. Ensure you entered the right GitLab IP address in Bamboo under 'Trigger +IP addresses'. +1. Remember that GitLab only triggers builds on push events. A commit via the +web interface will not trigger CI currently. diff --git a/doc/project_services/hipchat.md b/doc/project_services/hipchat.md new file mode 100644 index 00000000000..021a93a288f --- /dev/null +++ b/doc/project_services/hipchat.md @@ -0,0 +1,54 @@ +# Atlassian HipChat + +GitLab provides a way to send HipChat notifications upon a number of events, +such as when a user pushes code, creates a branch or tag, adds a comment, and +creates a merge request. + +## Setup + +GitLab requires the use of a HipChat v2 API token to work. v1 tokens are +not supported at this time. Note the differences between v1 and v2 tokens: + +HipChat v1 API (legacy) supports "API Auth Tokens" in the Group API menu. A v1 +token is allowed to send messages to *any* room. + +HipChat v2 API has tokens that are can be created using the Integrations tab +in the Group or Room admin page. By design, these are lightweight tokens that +allow GitLab to send messages only to *one* room. + +### Complete these steps in HipChat: + +1. Go to: https://admin.hipchat.com/admin +1. Click on "Group Admin" -> "Integrations". +1. Find "Build Your Own!" and click "Create". +1. Select the desired room, name the integration "GitLab", and click "Create". +1. In the "Send messages to this room by posting this URL" column, you should +see a URL in the format: + +``` + https://api.hipchat.com/v2/room/<room>/notification?auth_token=<token> +``` + +HipChat is now ready to accept messages from GitLab. Next, set up the HipChat +service in GitLab. + +### Complete these steps in GitLab: + +1. Navigate to the project you want to configure for notifications. +1. Select "Settings" in the top navigation. +1. Select "Services" in the left navigation. +1. Click "HipChat". +1. Select the "Active" checkbox. +1. Insert the `token` field from the URL into the `Token` field on the Web page. +1. Insert the `room` field from the URL into the `Room` field on the Web page. +1. Save or optionally click "Test Settings". + +## Troubleshooting + +If you do not see notifications, make sure you are using a HipChat v2 API +token, not a v1 token. + +Note that the v2 token is tied to a specific room. If you want to be able to +specify arbitrary rooms, you can create an API token for a specific user in +HipChat under "Account settings" and "API access". Use the `XXX` value under +`auth_token=XXX`. diff --git a/doc/project_services/irker.md b/doc/project_services/irker.md new file mode 100644 index 00000000000..780a45bca20 --- /dev/null +++ b/doc/project_services/irker.md @@ -0,0 +1,46 @@ +# Irker IRC Gateway + +GitLab provides a way to push update messages to an Irker server. When +configured, pushes to a project will trigger the service to send data directly +to the Irker server. + +See the project homepage for further info: http://www.catb.org/esr/irker/ + +## Needed setup + +You will first need an Irker daemon. You can download the Irker code from its +gitorious repository on https://gitorious.org/irker: `git clone +git@gitorious.org:irker/irker.git`. Once you have downloaded the code, you can +run the python script named `irkerd`. This script is the gateway script, it acts +both as an IRC client, for sending messages to an IRC server obviously, and as a +TCP server, for receiving messages from the GitLab service. + +If the Irker server runs on the same machine, you are done. If not, you will +need to follow the firsts steps of the next section. + +## Optional setup + +In the `app/models/project_services/irker_service.rb` file, you can modify some +options in the `initialize_settings` method: +- **server_ip** (defaults to `localhost`): the server IP address where the +`irkerd` daemon runs; +- **server_port** (defaults to `6659`): the server port of the `irkerd` daemon; +- **max_channels** (defaults to `3`): the maximum number of recipients the +client is authorized to join, per project; +- **default_irc_uri** (no default) : if this option is set, it has to be in the +format `irc[s]://domain.name` and will be prepend to each and every channel +provided by the user which is not a full URI. + +If the Irker server and the GitLab application do not run on the same host, you +will **need** to setup at least the **server_ip** option. + +## Note on Irker recipients + +Irker accepts channel names of the form `chan` and `#chan`, both for the +`#chan` channel. If you want to send messages in query, you will need to add +`,isnick` avec the channel name, in this form: `Aorimn,isnick`. In this latter +case, `Aorimn` is treated as a nick and no more as a channel name. + +Irker can also join password-protected channels. Users need to append +`?key=thesecretpassword` to the chan name. + diff --git a/doc/project_services/project_services.md b/doc/project_services/project_services.md new file mode 100644 index 00000000000..03937d20728 --- /dev/null +++ b/doc/project_services/project_services.md @@ -0,0 +1,20 @@ +# Project Services + +__Project integrations with external services for continuous integration and more.__ + +## Services + +- Assembla +- [Atlassian Bamboo CI](bamboo.md) An Atlassian product for continuous integration. +- Build box +- Campfire +- Emails on push +- Flowdock +- Gemnasium +- GitLab CI +- [HipChat](hipchat.md) An Atlassian product for private group chat and instant messaging. +- [Irker](irker.md) An IRC gateway to receive messages on repository updates. +- Pivotal Tracker +- Pushover +- Slack +- TeamCity diff --git a/doc/public_access/public_access.md b/doc/public_access/public_access.md index 4712c387021..7c5a6c04639 100644 --- a/doc/public_access/public_access.md +++ b/doc/public_access/public_access.md @@ -41,4 +41,4 @@ When visiting the public page of an user, you will only see listed projects whic ## Restricting the use of public or internal projects -In [gitlab.yml](https://gitlab.com/gitlab-org/gitlab-ce/blob/dbd88d453b8e6c78a423fa7e692004b1db6ea069/config/gitlab.yml.example#L64) you can disable public projects or public and internal projects for the entire GitLab installation to prevent people making code public by accident. +In [gitlab.yml](https://gitlab.com/gitlab-org/gitlab-ce/blob/dbd88d453b8e6c78a423fa7e692004b1db6ea069/config/gitlab.yml.example#L64) you can disable public projects or public and internal projects for the entire GitLab installation to prevent people making code public by accident. The restricted visibility settings do not apply to admin users. diff --git a/doc/raketasks/README.md b/doc/raketasks/README.md index 9e2f697bca6..770b7a70fe0 100644 --- a/doc/raketasks/README.md +++ b/doc/raketasks/README.md @@ -1,5 +1,8 @@ +# Rake tasks + - [Backup restore](backup_restore.md) - [Cleanup](cleanup.md) +- [Features](features.md) - [Maintenance](maintenance.md) and self-checks - [User management](user_management.md) - [Web hooks](web_hooks.md) diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index b4581e2a07a..99cdfff0ac6 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -13,8 +13,8 @@ You can only restore a backup to exactly the same version of GitLab that you cre # use this command if you've installed GitLab with the Omnibus package sudo gitlab-rake gitlab:backup:create -# if you've installed GitLab from source or using the cookbook -bundle exec rake gitlab:backup:create RAILS_ENV=production +# if you've installed GitLab from source +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ``` Example output: @@ -137,7 +137,7 @@ with the name of your bucket: Please be informed that a backup does not store your configuration files. If you use an Omnibus package please see the [instructions in the readme to backup your configuration](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#backup-and-restore-omnibus-gitlab-configuration). If you have a cookbook installation there should be a copy of your configuration in Chef. -If you have a manual installation please consider backing up your gitlab.yml file and any SSL keys and certificates. +If you have an installation from source, please consider backing up your `gitlab.yml` file, any SSL keys and certificates, and your [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). ## Restore a previously created backup @@ -147,7 +147,7 @@ You can only restore a backup to exactly the same version of GitLab that you cre # Omnibus package installation sudo gitlab-rake gitlab:backup:restore -# installation from source or cookbook +# installation from source bundle exec rake gitlab:backup:restore RAILS_ENV=production ``` @@ -192,7 +192,7 @@ Deleting tmp directories...[DONE] For Omnibus package installations, see https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#scheduling-a-backup . -For installation from source or cookbook: +For installation from source: ``` cd /home/git/gitlab sudo -u git -H editor config/gitlab.yml # Enable keep_time in the backup section to automatically delete old backups @@ -203,5 +203,31 @@ Add the following lines at the bottom: ``` # Create a full backup of the GitLab repositories and SQL database every day at 4am -0 4 * * * cd /home/git/gitlab && PATH=/usr/local/bin:/usr/bin:/bin bundle exec rake gitlab:backup:create RAILS_ENV=production +0 4 * * * cd /home/git/gitlab && PATH=/usr/local/bin:/usr/bin:/bin bundle exec rake gitlab:backup:create RAILS_ENV=production CRON=1 ``` + +The `CRON=1` environment setting tells the backup script to suppress all progress output if there are no errors. +This is recommended to reduce cron spam. + +## Alternative backup strategies + +If your GitLab server contains a lot of Git repository data you may find the GitLab backup script to be too slow. +In this case you can consider using filesystem snapshots as part of your backup strategy. + +Example: Amazon EBS + +> A GitLab server using omnibus-gitlab hosted on Amazon AWS. +> An EBS drive containing an ext4 filesystem is mounted at `/var/opt/gitlab`. +> In this case you could make an application backup by taking an EBS snapshot. +> The backup includes all repositories, uploads and Postgres data. + +Example: LVM snapshots + rsync + +> A GitLab server using omnibus-gitlab, with an LVM logical volume mounted at `/var/opt/gitlab`. +> Replicating the `/var/opt/gitlab` directory using rsync would not be reliable because too many files would change while rsync is running. +> Instead of rsync-ing `/var/opt/gitlab`, we create a temporary LVM snapshot, which we mount as a read-only filesystem at `/mnt/gitlab_backup`. +> Now we can have a longer running rsync job which will create a consistent replica on the remote server. +> The replica includes all repositories, uploads and Postgres data. + +If you are running GitLab on a virtualized server you can possibly also create VM snapshots of the entire GitLab server. +It is not uncommon however for a VM snapshot to require you to power down the server, so this approach is probably of limited practical use. diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index 9e48f56c951..96d67f7b5d6 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -8,7 +8,7 @@ Remove namespaces(dirs) from `/home/git/repositories` if they don't exist in Git # omnibus-gitlab sudo gitlab-rake gitlab:cleanup:dirs -# installation from source or cookbook +# installation from source bundle exec rake gitlab:cleanup:dirs RAILS_ENV=production ``` @@ -18,6 +18,6 @@ Remove repositories (global only for now) from `/home/git/repositories` if they # omnibus-gitlab sudo gitlab-rake gitlab:cleanup:repos -# installation from source or cookbook +# installation from source bundle exec rake gitlab:cleanup:repos RAILS_ENV=production ``` diff --git a/doc/raketasks/features.md b/doc/raketasks/features.md index 99b3d5525b0..f9a46193547 100644 --- a/doc/raketasks/features.md +++ b/doc/raketasks/features.md @@ -6,7 +6,7 @@ This command will enable the namespaces feature introduced in v4.0. It will move Note: -- Because the **repository location will change**, you will need to **update all your git url's** to point to the new location. +- Because the **repository location will change**, you will need to **update all your git URLs** to point to the new location. - Username can be changed at [Profile / Account](/profile/account) **Example:** diff --git a/doc/raketasks/import.md b/doc/raketasks/import.md index 39b1a52a44d..8a38937062e 100644 --- a/doc/raketasks/import.md +++ b/doc/raketasks/import.md @@ -1,28 +1,62 @@ -# Import +# Import bare repositories into your GitLab instance -### Import bare repositories into GitLab project instance +## Notes -Notes: +- The owner of the project will be the first admin +- The groups will be created as needed +- The owner of the group will be the first admin +- Existing projects will be skipped -* project owner will be a first admin -* groups will be created as needed -* group owner will be the first admin -* existing projects will be skipped +## How to use -How to use: +### Create a new folder inside the git repositories path. This will be the name of the new group. -1. copy your bare repos under git repos_path (see `config/gitlab.yml` gitlab_shell -> repos_path) -2. run the command below +- For omnibus-gitlab, it is located at: `/var/opt/gitlab/git-data/repositories` by default, unless you changed +it in the `/etc/gitlab/gitlab.rb` file. +- For installations from source, it is usually located at: `/home/git/repositories` or you can see where +your repositories are located by looking at `config/gitlab.yml` under the `gitlab_shell => repos_path` entry. + +New folder needs to have git user ownership and read/write/execute access for git user and its group: + +``` +sudo -u git mkdir /var/opt/gitlab/git-data/repositories/new_group +``` + +If you are using an installation from source, replace `/var/opt/gitlab/git-data` +with `/home/git`. + +### Copy your bare repositories inside this newly created folder: + +``` +sudo cp -r /old/git/foo.git /var/opt/gitlab/git-data/repositories/new_group/ + +# Do this once when you are done copying git repositories +sudo chown -R git:git /var/opt/gitlab/git-data/repositories/new_group/ +``` + +`foo.git` needs to be owned by the git user and git users group. + +If you are using an installation from source, replace `/var/opt/gitlab/git-data` +with `/home/git`. + +### Run the command below depending on your type of installation: + +#### Omnibus Installation ``` -# omnibus-gitlab -sudo gitlab-rake gitlab:import:repos +$ sudo gitlab-rake gitlab:import:repos +``` + +#### Installation from source -# installation from source or cookbook -bundle exec rake gitlab:import:repos RAILS_ENV=production +Before running this command you need to change the directory to where your GitLab installation is located: + +``` +$ cd /home/git/gitlab +$ sudo -u git -H bundle exec rake gitlab:import:repos RAILS_ENV=production ``` -Example output: +#### Example output ``` Processing abcd.git diff --git a/doc/raketasks/maintenance.md b/doc/raketasks/maintenance.md index f6bd7565799..41a994f3f68 100644 --- a/doc/raketasks/maintenance.md +++ b/doc/raketasks/maintenance.md @@ -8,7 +8,7 @@ This command gathers information about your GitLab installation and the System i # omnibus-gitlab sudo gitlab-rake gitlab:env:info -# installation from source or cookbook +# installation from source bundle exec rake gitlab:env:info RAILS_ENV=production ``` @@ -16,30 +16,31 @@ Example output: ``` System information -System: Debian 6.0.7 -Current User: git -Using RVM: no -Ruby Version: 2.0.0-p481 -Gem Version: 1.8.23 -Bundler Version:1.3.5 -Rake Version: 10.0.4 +System: Debian 7.8 +Current User: git +Using RVM: no +Ruby Version: 2.1.5p273 +Gem Version: 2.4.3 +Bundler Version: 1.7.6 +Rake Version: 10.3.2 +Sidekiq Version: 2.17.8 GitLab information -Version: 5.1.0.beta2 -Revision: 4da8b37 -Directory: /home/git/gitlab -DB Adapter: mysql2 -URL: http://example.com -HTTP Clone URL: http://example.com/some-project.git -SSH Clone URL: git@example.com:some-project.git -Using LDAP: no -Using Omniauth: no +Version: 7.7.1 +Revision: 41ab9e1 +Directory: /home/git/gitlab +DB Adapter: postgresql +URL: https://gitlab.example.com +HTTP Clone URL: https://gitlab.example.com/some-project.git +SSH Clone URL: git@gitlab.example.com:some-project.git +Using LDAP: no +Using Omniauth: no GitLab Shell -Version: 1.2.0 -Repositories: /home/git/repositories/ -Hooks: /home/git/gitlab-shell/hooks/ -Git: /usr/bin/git +Version: 2.4.1 +Repositories: /home/git/repositories/ +Hooks: /home/git/gitlab-shell/hooks/ +Git: /usr/bin/git ``` ## Check GitLab configuration @@ -59,7 +60,7 @@ You may also have a look at our [Trouble Shooting Guide](https://github.com/gitl # omnibus-gitlab sudo gitlab-rake gitlab:check -# installation from source or cookbook +# installation from source bundle exec rake gitlab:check RAILS_ENV=production ``` @@ -122,3 +123,56 @@ sudo -u git -H mkdir -p /home/git/gitlab-satellites sudo -u git -H bundle exec rake gitlab:satellites:create RAILS_ENV=production sudo chmod u+rwx,g=rx,o-rwx /home/git/gitlab-satellites ``` + +## Rebuild authorized_keys file + +In some case it is necessary to rebuild the `authorized_keys` file. + +For Omnibus-packages: +``` +sudo gitlab-rake gitlab:shell:setup +``` + +For installations from source: +``` +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:shell:setup RAILS_ENV=production +``` + +``` +This will rebuild an authorized_keys file. +You will lose any data stored in authorized_keys file. +Do you want to continue (yes/no)? yes +``` + +## Clear redis cache + +If for some reason the dashboard shows wrong information you might want to +clear Redis' cache. + +For Omnibus-packages: +``` +sudo gitlab-rake cache:clear +``` + +For installations from source: +``` +cd /home/git/gitlab +sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production +``` + +## Precompile the assets + +Sometimes during version upgrades you might end up with some wrong CSS or +missing some icons. In that case, try to precompile the assets again. + +For Omnibus-packages: +``` +sudo gitlab-rake assets:precompile +``` + +For installations from source: +``` +cd /home/git/gitlab +sudo -u git -H bundle exec rake assets:precompile RAILS_ENV=production +``` diff --git a/doc/raketasks/user_management.md b/doc/raketasks/user_management.md index 3c67753ad28..80b01ca4043 100644 --- a/doc/raketasks/user_management.md +++ b/doc/raketasks/user_management.md @@ -6,7 +6,7 @@ # omnibus-gitlab sudo gitlab-rake gitlab:import:user_to_projects[username@domain.tld] -# installation from source or cookbook +# installation from source bundle exec rake gitlab:import:user_to_projects[username@domain.tld] RAILS_ENV=production ``` @@ -20,7 +20,7 @@ Notes: # omnibus-gitlab sudo gitlab-rake gitlab:import:all_users_to_all_projects -# installation from source or cookbook +# installation from source bundle exec rake gitlab:import:all_users_to_all_projects RAILS_ENV=production ``` @@ -30,7 +30,7 @@ bundle exec rake gitlab:import:all_users_to_all_projects RAILS_ENV=production # omnibus-gitlab sudo gitlab-rake gitlab:import:user_to_groups[username@domain.tld] -# installation from source or cookbook +# installation from source bundle exec rake gitlab:import:user_to_groups[username@domain.tld] RAILS_ENV=production ``` @@ -44,6 +44,6 @@ Notes: # omnibus-gitlab sudo gitlab-rake gitlab:import:all_users_to_all_groups -# installation from source or cookbook +# installation from source bundle exec rake gitlab:import:all_users_to_all_groups RAILS_ENV=production ``` diff --git a/doc/raketasks/web_hooks.md b/doc/raketasks/web_hooks.md index e1a58835d88..5a8b94af9b4 100644 --- a/doc/raketasks/web_hooks.md +++ b/doc/raketasks/web_hooks.md @@ -4,42 +4,42 @@ # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" - # source installations or cookbook + # source installations bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" RAILS_ENV=production ## Add a web hook for projects in a given **NAMESPACE**: # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme - # source installations or cookbook + # source installations bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production ## Remove a web hook from **ALL** projects using: # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" - # source installations or cookbook + # source installations bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" RAILS_ENV=production ## Remove a web hook from projects in a given **NAMESPACE**: # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme - # source installations or cookbook + # source installations bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production ## List **ALL** web hooks: # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:list - # source installations or cookbook + # source installations bundle exec rake gitlab:web_hook:list RAILS_ENV=production ## List the web hooks from projects in a given **NAMESPACE**: # omnibus-gitlab sudo gitlab-rake gitlab:web_hook:list NAMESPACE=/ - # source installations or cookbook + # source installations bundle exec rake gitlab:web_hook:list NAMESPACE=/ RAILS_ENV=production > Note: `/` is the global namespace. diff --git a/doc/release/howto_rc1.md b/doc/release/howto_rc1.md new file mode 100644 index 00000000000..07c703142d4 --- /dev/null +++ b/doc/release/howto_rc1.md @@ -0,0 +1,55 @@ +# How to create RC1 + +The RC1 release comes with the task to update the installation and upgrade docs. Be mindful that there might already be merge requests for this on GitLab or GitHub. + +### 1. Update the installation guide + +1. Check if it references the correct branch `x-x-stable` (doesn't exist yet, but that is okay) +1. Check the [GitLab Shell version](/lib/tasks/gitlab/check.rake#L782) +1. Check the [Git version](/lib/tasks/gitlab/check.rake#L794) +1. There might be other changes. Ask around. + +### 2. Create update guides + +[Follow this guide](howto_update_guides.md) to create update guides. + +### 3. Code quality indicators + +Make sure the code quality indicators are green / good. + +- [](http://ci.gitlab.org/projects/1?ref=master) on ci.gitlab.org (master branch) + +- [](https://semaphoreapp.com/gitlabhq/gitlabhq) (master branch) + +- [](https://codeclimate.com/github/gitlabhq/gitlabhq) + +- [](https://gemnasium.com/gitlabhq/gitlabhq) this button can be yellow (small updates are available) but must not be red (a security fix or an important update is available) + +- [](https://coveralls.io/r/gitlabhq/gitlabhq) + +### 4. Run release tool + +**Make sure EE `master` has latest changes from CE `master`** + +Get release tools + +``` +git clone git@dev.gitlab.org:gitlab/release-tools.git +cd release-tools +``` + +Release candidate creates stable branch from master. +So we need to sync master branch between all CE, EE and CI remotes. + +``` +bundle exec rake sync +``` + +Create release candidate and stable branch: + +``` +bundle exec rake release["x.x.0.rc1"] +``` + +Now developers can use master for merging new features. +So you should use stable branch for future code changes related to release. diff --git a/doc/release/howto_update_guides.md b/doc/release/howto_update_guides.md new file mode 100644 index 00000000000..23d0959c33d --- /dev/null +++ b/doc/release/howto_update_guides.md @@ -0,0 +1,55 @@ +# Create update guides + +1. Create: CE update guide from previous version. Like `7.3-to-7.4.md` +1. Create: CE to EE update guide in EE repository for latest version. +1. Update: `6.x-or-7.x-to-7.x.md` to latest version. +1. Create: CI update guide from previous version + +It's best to copy paste the previous guide and make changes where necessary. +The typical steps are listed below with any points you should specifically look at. + +#### 0. Any major changes? + +List any major changes here, so the user is aware of them before starting to upgrade. For instance: + +- Database updates +- Web server changes +- File structure changes + +#### 1. Stop server + +#### 2. Make backup + +#### 3. Do users need to update dependencies like `git`? + +- Check if the [GitLab Shell version](/lib/tasks/gitlab/check.rake#L782) changed since the last release. + +- Check if the [Git version](/lib/tasks/gitlab/check.rake#L794) changed since the last release. + +#### 4. Get latest code + +#### 5. Does GitLab shell need to be updated? + +#### 6. Install libs, migrations, etc. + +#### 7. Any config files updated since last release? + +Check if any of these changed since last release: + +- [lib/support/nginx/gitlab](/lib/support/nginx/gitlab) +- [lib/support/nginx/gitlab-ssl](/lib/support/nginx/gitlab-ssl) +- <https://gitlab.com/gitlab-org/gitlab-shell/commits/master/config.yml.example> +- [config/gitlab.yml.example](/config/gitlab.yml.example) +- [config/unicorn.rb.example](/config/unicorn.rb.example) +- [config/database.yml.mysql](/config/database.yml.mysql) +- [config/database.yml.postgresql](/config/database.yml.postgresql) +- [config/initializers/rack_attack.rb.example](/config/initializers/rack_attack.rb.example) +- [config/resque.yml.example](/config/resque.yml.example) + +#### 8. Need to update init script? + +Check if the `init.d/gitlab` script changed since last release: [lib/support/init.d/gitlab](/lib/support/init.d/gitlab) + +#### 9. Start application + +#### 10. Check application status diff --git a/doc/release/monthly.md b/doc/release/monthly.md index c46a3ed9c93..cfe01896d8f 100644 --- a/doc/release/monthly.md +++ b/doc/release/monthly.md @@ -1,200 +1,126 @@ # Monthly Release -NOTE: This is a guide for GitLab developers. +NOTE: This is a guide used by the GitLab B.V. developers. -# **15th - Code Freeze & Release Manager** +It starts 7 working days before the release. +The release manager doesn't have to perform all the work but must ensure someone is assigned. +The current release manager must schedule the appointment of the next release manager. +The new release manager should create overall issue to track the progress. -### **1. Stop merging in code, except for important bugfixes** +## Release Manager -### **2. Release Manager** +A release manager is selected that coordinates all releases the coming month, including the patch releases for previous releases. +The release manager has to make sure all the steps below are done and delegated where necessary. +This person should also make sure this document is kept up to date and issues are created and updated. -A release manager is selected that coordinates the entire release of this version. The release manager has to make sure all the steps below are done and delegated where necessary. This person should also make sure this document is kept up to date and issues are created and updated. +## Take vacations into account -### **3. Create an overall issue** -Name it "Release x.x.x" for easier searching. +The time is measured in weekdays to compensate for weekends. +Do everything on time to prevent problems due to rush jobs or too little testing time. +Make sure that you take into account any vacations of maintainers. +If the release is falling behind immediately warn the team. -``` -15th: - -* Update the changelog (#LINK) -* Triage the omnibus-gitlab milestone - -16th: - -* Merge CE in to EE (#LINK) -* Close the omnibus-gitlab milestone - -17th: - -* Create x.x.0.rc1 (#LINK) -* Build package for GitLab.com (https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/administration.md#build-a-package) +## Create an overall issue and follow it -18th: - -* Update GitLab.com with rc1 (#LINK) (https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/administration.md#deploy-the-package) -* Regression issue and tweet about rc1 (#LINK) -* Start blog post (#LINK) - -21th: - -* Do QA and fix anything coming out of it (#LINK) - -22nd: +Create issue for GitLab CE project(internal). Name it "Release x.x.x" for easier searching. +Replace the dates with actual dates based on the number of workdays before the release. +All steps from issue template are explained below -* Release CE and EE (#LINK) - -23rd: - -* Prepare package for GitLab.com release (#LINK) - -24th: - -* Deploy to GitLab.com (#LINK) ``` +Xth: (7 working days before the 22nd) -### **4. Update Changelog** - -Any changes not yet added to the changelog are added by lead developer and in that merge request the complete team is asked if there is anything missing. - -### **5. Take weekend and vacations into account** - -Ensure that there is enough time to incorporate the findings of the release candidate, etc. - -# **16th - Merge the CE into EE** - -Do this via a merge request. - -# **17th - Create RC1** +- [ ] Code freeze +- [ ] Update the CE changelog (#LINK) +- [ ] Update the EE changelog (#LINK) +- [ ] Update the CI changelog (#LINK) +- [ ] Triage the omnibus-gitlab milestone -The RC1 release comes with the task to update the installation and upgrade docs. Be mindful that there might already be merge requests for this on GitLab or GitHub. +Xth: (6 working days before the 22nd) -### **1. Update the installation guide** +- [ ] Merge CE master in to EE master via merge request (#LINK) +- [ ] Determine QA person and notify this person +- [ ] Check the tasks in [how to rc1 guide](howto_rc1.md) and delegate tasks if necessary +- [ ] Create CE, EE, CI RC1 versions (#LINK) -1. Check if it references the correct branch `x-x-stable` (doesn't exist yet, but that is okay) -1. Check the [GitLab Shell version](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/tasks/gitlab/check.rake#L782) -1. Check the [Git version](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/tasks/gitlab/check.rake#L794) -1. There might be other changes. Ask around. +Xth: (5 working days before the 22nd) -### **2. Create an update guides** +- [ ] Do QA and fix anything coming out of it (#LINK) +- [ ] Close the omnibus-gitlab milestone +- [ ] Prepare the blog post (#LINK) -1. Create: CE update guide from previous version. Like `from-6-8-to-6.9` -1. Create: CE to EE update guide in EE repository for latest version. -1. Update: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/6.0-to-6.x.md to latest version. +Xth: (4 working days before the 22nd) -It's best to copy paste the previous guide and make changes where necessary. -The typical steps are listed below with any points you should specifically look at. +- [ ] Update GitLab.com with rc1 (#LINK) (https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/administration.md#deploy-the-package) +- [ ] Update ci.gitLab.com with rc1 (#LINK) (https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/administration.md#deploy-the-package) +- [ ] Create regression issues (CE, CI) (#LINK) +- [ ] Tweet about rc1 (#LINK) -#### 0. Any major changes? +Xth: (3 working days before the 22nd) -List any major changes here, so the user is aware of them before starting to upgrade. For instance: +- [ ] Merge CE stable branch into EE stable branch -- Database updates -- Web server changes -- File structure changes +Xth: (2 working days before the 22nd) -#### 1. Make backup +- [ ] Check that everyone is mentioned on the blog post (the reviewer should have done this one working day ago) +- [ ] Check that MVP is added to the mvp page (source/mvp/index.html in www-gitlab-com) -#### 2. Stop server +Xth: (1 working day before the 22nd) -#### 3. Do users need to update dependencies like `git`? +- [ ] Create CE, EE, CI stable versions (#LINK) +- [ ] Create Omnibus tags and build packages +- [ ] Update GitLab.com with the stable version (#LINK) +- [ ] Update ci.gitLab.com with the stable version (#LINK) -- Check if the [GitLab Shell version](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/tasks/gitlab/check.rake#L782) changed since the last release. - -- Check if the [Git version](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/tasks/gitlab/check.rake#L794) changed since the last release. - -#### 4. Get latest code - -#### 5. Does GitLab shell need to be updated? - -#### 6. Install libs, migrations, etc. - -#### 7. Any config files updated since last release? - -Check if any of these changed since last release: - -- <https://gitlab.com/gitlab-org/gitlab-ce/commits/master/lib/support/nginx/gitlab> -- <https://gitlab.com/gitlab-org/gitlab-ce/commits/master/lib/support/nginx/gitlab-ssl> -- <https://gitlab.com/gitlab-org/gitlab-shell/commits/master/config.yml.example> -- <https://gitlab.com/gitlab-org/gitlab-ce/commits/master/config/gitlab.yml.example> -- <https://gitlab.com/gitlab-org/gitlab-ce/commits/master/config/unicorn.rb.example> -- <https://gitlab.com/gitlab-org/gitlab-ce/commits/master/config/database.yml.mysql> -- <https://gitlab.com/gitlab-org/gitlab-ce/commits/master/config/database.yml.postgresql> -- <https://gitlab.com/gitlab-org/gitlab-ce/commits/master/config/initializers/rack_attack.rb.example> -- <https://gitlab.com/gitlab-org/gitlab-ce/commits/master/config/resque.yml.example> - -#### 8. Need to update init script? - -Check if the `init.d/gitlab` script changed since last release: <https://gitlab.com/gitlab-org/gitlab-ce/commits/master/lib/support/init.d/gitlab> - -#### 9. Start application - -#### 10. Check application status +22nd: -### **3. Code quality indicators** +- [ ] Release CE, EE and CI (#LINK) -Make sure the code quality indicators are green / good. +``` -- [](http://ci.gitlab.org/projects/1?ref=master) on ci.gitlab.org (master branch) +- - - -- [](https://semaphoreapp.com/gitlabhq/gitlabhq) (master branch) +## Code Freeze -- [](https://codeclimate.com/github/gitlabhq/gitlabhq) +Stop merging code in master, except for important bug fixes -- [](https://gemnasium.com/gitlabhq/gitlabhq) this button can be yellow (small updates are available) but must not be red (a security fix or an important update is available) +## Update changelog -- [](https://coveralls.io/r/gitlabhq/gitlabhq) +Any changes not yet added to the changelog are added by lead developer and in that merge request the complete team is +asked if there is anything missing. -### **4. Set VERSION** +There are three changelogs that need to be updated: CE, EE and CI. -Change version in VERSION to `x.x.0.rc1`. +## Create RC1 (CE, EE, CI) -### **5. Tag** +[Follow this How-to guide](howto_rc1.md) to create RC1. -Create an annotated tag that points to the version change commit: +## Prepare CHANGELOG for next release -``` -git tag -a vx.x.0.rc1 -m 'Version x.x.0.rc1' -``` +Once the stable branches have been created, update the CHANGELOG in `master` with the upcoming version, usually X.X.X.pre. -### **6. Create stable branches** +## QA -For GitLab EE, append `-ee` to the branch. +Create issue on dev.gitlab.org `gitlab` repository, named "GitLab X.X QA" in order to keep track of the progress. -`x-x-stable-ee` +Use the omnibus packages created for RC1 of Enterprise Edition using [this guide](https://dev.gitlab.org/gitlab/gitlab-ee/blob/master/doc/release/manual_testing.md). -``` -git checkout master -git pull -git checkout -b x-x-stable -git push <remote> x-x-stable -``` +**NOTE** Upgrader can only be tested when tags are pushed to all repositories. Do not forget to confirm it is working before releasing. Note that in the issue. -Now developers can use master for merging new features. -So you should use stable branch for future code chages related to release. +#### Fix anything coming out of the QA +Create an issue with description of a problem, if it is quick fix fix it yourself otherwise contact the team for advice. -# **18th - Release RC1** +**NOTE** If there is a problem that cannot be fixed in a timely manner, reverting the feature is an option! If the feature is reverted, +create an issue about it in order to discuss the next steps after the release. -### **1. Update GitLab.com** +## Update GitLab.com with RC1 -Merge the RC1 EE code into GitLab.com. -Once the build is green, create a package. +Use the omnibus EE packages created for RC1. If there are big database migrations consider testing them with the production db on a VM. Try to deploy in the morning. It is important to do this as soon as possible, so we can catch any errors before we release the full version. -### **2. Prepare the blog post** - -- Start with a complete copy of the [release blog template](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/doc/release_blog_template.md) and fill it out. -- Check the changelog of CE and EE for important changes. -- Create a WIP MR for the blog post -- Ask Dmitriy to add screenshots to the WIP MR. -- Decide with team who will be the MVP user. -- Add a note if there are security fixes: This release fixes an important security issue and we advise everyone to upgrade as soon as possible. -- Assign to one reviewer who will fix spelling issues by editing the branch (can use the online editor) -- After the reviewer is finished the whole team will be mentioned to give their suggestions via line comments - -### **3. Create a regressions issue** +## Create a regressions issue On [the GitLab CE issue tracker on GitLab.com](https://gitlab.com/gitlab-org/gitlab-ce/issues/) create an issue titled "GitLab X.X regressions" add the following text: @@ -203,122 +129,84 @@ Please do not raise issues directly in this issue but link to issues that might The decision to create a patch release or not is with the release manager who is assigned to this issue. The release manager will comment here about the plans for patch releases. -Assign the issue to the release manager and /cc all the core-team members active on the issue tracker. If there are any known bugs in the release add them immediately. +Assign the issue to the release manager and at mention all members of gitlab core team. If there are any known bugs in the release add them immediately. -### **4. Tweet** +## Tweet about RC1 Tweet about the RC release: > GitLab x.x.0.rc1 is out. This release candidate is only suitable for testing. Please link regressions issues from LINK_TO_REGRESSION_ISSUE -# **21st - Preparation** - -### **1. Pre QA merge** - -Merge CE into EE before doing the QA. - -### **2. QA** - -Create issue on dev.gitlab.org `gitlab` repository, named "GitLab X.X QA" in order to keep track of the progress. - -Use the omnibus packages of Enterprise Edition using [this guide](https://dev.gitlab.org/gitlab/gitlab-ee/blob/master/doc/release/manual_testing.md). - -**NOTE** Upgrader can only be tested when tags are pushed to all repositories. Do not forget to confirm it is working before releasing. Note that in the issue. - -### **3. Fix anything coming out of the QA** - -Create an issue with description of a problem, if it is quick fix fix it yourself otherwise contact the team for advice. - -**NOTE** If there is a problem that cannot be fixed in a timely manner, reverting the feature is an option! If the feature is reverted, -create an issue about it in order to discuss the next steps after the release. - -# **22nd - Release CE and EE** - -For GitLab EE, append `-ee` to the branches and tags. +## Prepare the blog post -`x-x-stable-ee` +1. Start with a complete copy of the [release blog template](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/doc/release_blog_template.md) and fill it out. +1. Make sure the blog post contains information about the GitLab CI release. +1. Check the changelog of CE and EE for important changes. +1. Also check the CI changelog +1. Add a proposed tweet text to the blog post WIP MR description. +1. Create a WIP MR for the blog post +1. Ask Dmitriy (or a team member with OS X) to add screenshots to the WIP MR. +1. Decide with core team who will be the MVP user. +1. Create WIP MR for adding MVP to MVP page on website +1. Add a note if there are security fixes: This release fixes an important security issue and we advise everyone to upgrade as soon as possible. +1. Create a merge request on [GitLab.com](https://gitlab.com/gitlab-com/www-gitlab-com/tree/master) +1. Assign to one reviewer who will fix spelling issues by editing the branch (either with a git client or by using the online editor) +1. Comment to the reviewer: '@person Please mention the whole team as soon as you are done (3 workdays before release at the latest)' -`v.x.x.0-ee` +## Create CE, EE, CI stable versions -Note: Merge CE into EE if needed. - -### **1. Set VERSION to x.x.x and push** - -- Change the GITLAB_SHELL_VERSION file in `master` of the CE repository if the version changed. -- Change the GITLAB_SHELL_VERSION file in `master` of the EE repository if the version changed. -- Change the VERSION file in `master` branch of the CE repository and commit and push to origin. -- Change the VERSION file in `master` branch of the EE repository and commit and push to origin. - -### **2. Update installation.md** - -Update [installation.md](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md) to the newest version in master. - -### **3. Push latest changes from x-x-stable branch to dev.gitlab.org** +Get release tools ``` -git checkout -b x-x-stable -git push origin x-x-stable +git clone git@dev.gitlab.org:gitlab/release-tools.git +cd release-tools ``` -### **4. Build the Omnibus packages** - -Follow the [release doc in the Omnibus repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/release.md). -This can happen before tagging because Omnibus uses tags in its own repo and SHA1's to refer to the GitLab codebase. - -### **5. Create annotated tag vx.x.x** - -In `x-x-stable` branch check for the SHA-1 of the commit with VERSION file changed. Tag that commit, +Bump version, create release tag and push to remotes: ``` -git tag -a vx.x.0 -m 'Version x.x.0' xxxxx +bundle exec rake release["x.x.0"] ``` -where `xxxxx` is SHA-1. +This will create correct version and tag and push to all CE, EE and CI remotes. -### **6. Push the tag and x-x-stable branch to the remotes** +Update [installation.md](/doc/install/installation.md) to the newest version in master. -For GitLab CE, push to dev, GitLab.com and GitHub. -For GitLab EE, push to the subscribers repo. +## Create Omnibus tags and build packages -Make sure the branch is marked 'protected' on each of the remotes you pushed to. +Follow the [release doc in the Omnibus repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/release.md). +This can happen before tagging because Omnibus uses tags in its own repo and SHA1's to refer to the GitLab codebase. -``` -git push <remote> x-x-stable(-ee) -git push <remote> vx.x.0 -``` +## Update GitLab.com with the stable version + +- Deploy the package (should not need downtime because of the small difference with RC1) +- Deploy the package for ci.gitlab.com -### **7. Publish packages for new release** +## Release CE, EE and CI + +__1. Publish packages for new release__ Update `downloads/index.html` and `downloads/archive/index.html` in `www-gitlab-com` repository. -### **8. Publish blog for new release** +__2. Publish blog for new release__ +Doublecheck the everyone has been mentioned in the blog post. Merge the [blog merge request](#1-prepare-the-blog-post) in `www-gitlab-com` repository. -### **9. Tweet to blog** +__3. Tweet to blog__ Send out a tweet to share the good news with the world. List the most important features and link to the blog post. -Proposed tweet for CE "GitLab X.X is released! It brings *** <link-to-blogpost>" - -### **10. Send out the newsletter** - -Send out an email to the 'GitLab Newsletter' mailing list on MailChimp. -Replicate the former release newsletter and modify it accordingly. -**Do not forget to edit `Subject line` and regenerate `Plain-Text Email` from HTML source** - -Include a link to the blog post and keep it short. - -Proposed email text: -"We have released a new version of GitLab. See our blog post(<link>) for more information." +Proposed tweet "Release of GitLab X.X & CI Y.Y! FEATURE, FEATURE and FEATURE <link-to-blog-post> #gitlab" +Consider creating a post on Hacker News. -# **23rd - Optional Patch Release** +## Release new AMIs -# **24th - Update GitLab.com** +[Follow this guide](https://dev.gitlab.org/gitlab/AMI/blob/master/README.md) -Merge the stable release into GitLab.com. Once the build is green deploy the next morning. +## Create a WIP blogpost for the next release -# **25th - Release GitLab CI** +Create a WIP blogpost using [release blog template](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/doc/release_blog_template.md). diff --git a/doc/release/patch.md b/doc/release/patch.md index bcc14568fc8..68156ae9c0e 100644 --- a/doc/release/patch.md +++ b/doc/release/patch.md @@ -10,22 +10,49 @@ Otherwise include it in the monthly release and note there was a regression fix ## Release Procedure +### Preparation + 1. Verify that the issue can be reproduced 1. Note in the 'GitLab X.X regressions' that you will create a patch 1. Create an issue on private GitLab development server 1. Name the issue "Release X.X.X CE and X.X.X EE", this will make searching easier 1. Fix the issue on a feature branch, do this on the private GitLab development server +1. If it is a security issue, then assign it to the release manager and apply a 'security' label 1. Consider creating and testing workarounds 1. After the branch is merged into master, cherry pick the commit(s) into the current stable branch -1. In a separate commit in the stable branch update the CHANGELOG -1. For EE, update the CHANGELOG-EE if it is EE specific fix. Otherwise, merge the stable CE branch and add to CHANGELOG-EE "Merge community edition changes for version X.X.X" -1. In a separate commit in the stable branch update the VERSION -1. Create an annotated tag vX.X.X for CE and another patch release for EE `git tag -a vx.x.x -m 'Version x.x.x'` 1. Make sure that the build has passed and all tests are passing -1. Push the code and the tags to all the CE and EE repositories -1. Apply the patch to GitLab Cloud and the private GitLab development server +1. In a separate commit in the master branch update the CHANGELOG +1. For EE, update the CHANGELOG-EE if it is EE specific fix. Otherwise, merge the stable CE branch and add to CHANGELOG-EE "Merge community edition changes for version X.X.X" +1. Merge CE stable branch into EE stable branch + + +### Bump version + +Get release tools + +``` +git clone git@dev.gitlab.org:gitlab/release-tools.git +cd release-tools +``` + +Bump version in stable branch, create release tag and push to remotes: + +``` +bundle exec rake release["x.x.x"] +``` + +Or if you need to release only EE: + +``` +CE=false be rake release['x.x.x'] +``` + +### Release + 1. [Build new packages with the latest version](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/release.md) -1. Cherry-pick the changelog update back into master -1. Send tweets about the release from `@gitlabhq`, tweet should include the most important feature that the release is addressing as well as the link to the changelog +1. Apply the patch to GitLab.com and the private GitLab development server +1. Apply the patch to ci.gitLab.com and the private GitLab CI development server +1. Create and publish a blog post, see [patch release blog template](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/doc/patch_release_blog_template.md) +1. Send tweets about the release from `@gitlab`, tweet should include the most important feature that the release is addressing and link to the blog post 1. Note in the 'GitLab X.X regressions' issue that the patch was published (CE only) -1. Send out an email to the 'GitLab Newsletter' mailing list on MailChimp (or the 'Subscribers' list if the patch is EE only) +1. [Create new AMIs](https://dev.gitlab.org/gitlab/AMI/blob/master/README.md) diff --git a/doc/release/security.md b/doc/release/security.md index 79d23c02ea4..60bcfbb6da5 100644 --- a/doc/release/security.md +++ b/doc/release/security.md @@ -14,13 +14,17 @@ Please report suspected security vulnerabilities in private to <support@gitlab.c 1. Verify that the issue can be reproduced 1. Acknowledge the issue to the researcher that disclosed it +1. Inform the release manager that there needs to be a security release 1. Do the steps from [patch release document](doc/release/patch.md), starting with "Create an issue on private GitLab development server" +1. The MR with the security fix should get a 'security' label and be assigned to the release manager +1. Build the package for GitLab.com and do a deploy +1. Build the package for ci.gitLab.com and do a deploy +1. [Create new AMIs](https://dev.gitlab.org/gitlab/AMI/blob/master/README.md) 1. Create feature branches for the blog post on GitLab.com and link them from the code branch 1. Merge and publish the blog posts 1. Send tweets about the release from `@gitlabhq` -1. Send out an email to the 'GitLab Newsletter' mailing list on MailChimp (or the 'Subscribers' list if the security fix is for EE only) 1. Send out an email to [the community google mailing list](https://groups.google.com/forum/#!forum/gitlabhq) -1. Post a signed copy of our complete announcement to [oss-security](http://www.openwall.com/lists/oss-security/) and request a CVE number +1. Post a signed copy of our complete announcement to [oss-security](http://www.openwall.com/lists/oss-security/) and request a CVE number. CVE is only needed for bugs that allow someone to own the server (Remote Code Execution) or access to code of projects they are not a member of. 1. Add the security researcher to the [Security Researcher Acknowledgments list](http://about.gitlab.com/vulnerability-acknowledgements/) 1. Thank the security researcher in an email for their cooperation 1. Update the blog post and the CHANGELOG when we receive the CVE number diff --git a/doc/security/README.md b/doc/security/README.md index f88375f2afd..49dfa6eec76 100644 --- a/doc/security/README.md +++ b/doc/security/README.md @@ -2,4 +2,5 @@ - [Password length limits](password_length_limits.md) - [Rack attack](rack_attack.md) +- [Web Hooks and insecure internal web services](webhooks.md) - [Information exclusivity](information_exclusivity.md) diff --git a/doc/security/information_exclusivity.md b/doc/security/information_exclusivity.md index 127166ae2e7..f8e7fc3fd0e 100644 --- a/doc/security/information_exclusivity.md +++ b/doc/security/information_exclusivity.md @@ -4,6 +4,6 @@ Git is a distributed version control system (DVCS). This means that everyone that works with the source code has a local copy of the complete repository. In GitLab every project member that is not a guest (so reporters, developers and masters) can clone the repository to get a local copy. After obtaining this local copy the user can upload the full repository anywhere, including another project under their control or another server. -The consequense is that you can't build access controls that prevent the intentional sharing of source code by users that have access to the source code. +The consequence is that you can't build access controls that prevent the intentional sharing of source code by users that have access to the source code. This is an inherent feature of a DVCS and all git management systems have this limitation. Obviously you can take steps to prevent unintentional sharing and information destruction, this is why only some people are allowed to invite others and nobody can force push a protected branch. diff --git a/doc/security/webhooks.md b/doc/security/webhooks.md new file mode 100644 index 00000000000..1e9d33e87c3 --- /dev/null +++ b/doc/security/webhooks.md @@ -0,0 +1,13 @@ +# Web Hooks and insecure internal web services + +If you have non-GitLab web services running on your GitLab server or within its local network, these may be vulnerable to exploitation via Web Hooks. + +With [Web Hooks](../web_hooks/web_hooks.md), you and your project masters and owners can set up URLs to be triggered when specific things happen to projects. Normally, these requests are sent to external web services specifically set up for this purpose, that process the request and its attached data in some appropriate way. + +Things get hairy, however, when a Web Hook is set up with a URL that doesn't point to an external, but to an internal service, that may do something completely unintended when the web hook is triggered and the POST request is sent. + +Because Web Hook requests are made by the GitLab server itself, these have complete access to everything running on the server (http://localhost:123) or within the server's local network (http://192.168.1.12:345), even if these services are otherwise protected and inaccessible from the outside world. + +If a web service does not require authentication, Web Hooks can be used to trigger destructive commands by getting the GitLab server to make POST requests to endpoints like "http://localhost:123/some-resource/delete". + +To prevent this type of exploitation from happening, make sure that you are aware of every web service GitLab could potentially have access to, and that all of these are set up to require authentication for every potentially destructive command. Enabling authentication but leaving a default password is not enough.
\ No newline at end of file diff --git a/doc/ssh/README.md b/doc/ssh/README.md index c87fffd7d2c..6fe23dfa2a6 100644 --- a/doc/ssh/README.md +++ b/doc/ssh/README.md @@ -1,4 +1,73 @@ # SSH -- [Deploy keys](deploy_keys.md) -- [SSH](ssh.md) +## SSH keys + +An SSH key allows you to establish a secure connection between your +computer and GitLab. + +Before generating an SSH key, check if your system already has one by +running `cat ~/.ssh/id_rsa.pub`. If you see a long string starting with +`ssh-rsa` or `ssh-dsa`, you can skip the ssh-keygen step. + +To generate a new SSH key, just open your terminal and use code below. The +ssh-keygen command prompts you for a location and filename to store the key +pair and for a password. When prompted for the location and filename, you +can press enter to use the default. + +It is a best practice to use a password for an SSH key, but it is not +required and you can skip creating a password by pressing enter. Note that +the password you choose here can't be altered or retrieved. + +```bash +ssh-keygen -t rsa -C "$your_email" +``` + +Use the code below to show your public key. + +```bash +cat ~/.ssh/id_rsa.pub +``` + +Copy-paste the key to the 'My SSH Keys' section under the 'SSH' tab in your +user profile. Please copy the complete key starting with `ssh-` and ending +with your username and host. + +Use code below to copy your public key to the clipboard. Depending on your +OS you'll need to use a different command: + +**Windows:** +```bash +clip < ~/.ssh/id_rsa.pub +``` + +**Mac:** +```bash +pbcopy < ~/.ssh/id_rsa.pub +``` + +**Linux (requires xclip):** +```bash +xclip -sel clip < ~/.ssh/id_rsa.pub +``` + +## Deploy keys + +Deploy keys allow read-only access to multiple projects with a single SSH +key. + +This is really useful for cloning repositories to your Continuous +Integration (CI) server. By using deploy keys, you don't have to setup a +dummy user account. + +If you are a project master or owner, you can add a deploy key in the +project settings under the section 'Deploy Keys'. Press the 'New Deploy +Key' button and upload a public SSH key. After this, the machine that uses +the corresponding private key has read-only access to the project. + +You can't add the same deploy key twice with the 'New Deploy Key' option. +If you want to add the same key to another project, please enable it in the +list that says 'Deploy keys from projects available to you'. All the deploy +keys of all the projects you have access to are available. This project +access can happen through being a direct member of the projecti, or through +a group. See `def accessible_deploy_keys` in `app/models/user.rb` for more +information. diff --git a/doc/ssh/deploy_keys.md b/doc/ssh/deploy_keys.md deleted file mode 100644 index dcca8bdc61a..00000000000 --- a/doc/ssh/deploy_keys.md +++ /dev/null @@ -1,9 +0,0 @@ -# Deploy keys - -Deploy keys allow read-only access one or multiple projects with a single SSH key. - -This is really useful for cloning repositories to your Continuous Integration (CI) server. By using a deploy keys you don't have to setup a dummy user account. - -If you are a project master or owner you can add a deploy key in the project settings under the section Deploy Keys. Press the 'New Deploy Key' button and upload a public ssh key. After this the machine that uses the corresponding private key has read-only access to the project. - -You can't add the same deploy key twice with the 'New Deploy Key' option. If you want to add the same key to another project please enable it in the list that says 'Deploy keys from projects available to you'. All the deploy keys of all the projects you have access to are available. This project access can happen through being a direct member of the project or through a group. See `def accessible_deploy_keys` in `app/models/user.rb` for more information. diff --git a/doc/ssh/ssh.md b/doc/ssh/ssh.md deleted file mode 100644 index d466c1bde72..00000000000 --- a/doc/ssh/ssh.md +++ /dev/null @@ -1,21 +0,0 @@ -# SSH keys - -SSH key allows you to establish a secure connection between your computer and GitLab - -Before generating an SSH key, check if your system already has one by running `cat ~/.ssh/id_rsa.pub` If your see a long string starting with `ssh-rsa` or `ssh-dsa`, you can skip the ssh-keygen step. - -To generate a new SSH key just open your terminal and use code below. The ssh-keygen command prompts you for a location and filename to store the key pair and for a password. When prompted for the location and filename you can press enter to use the default. -It is a best practice to use a password for an SSH key but it is not required and you can skip creating a password by pressing enter. -Note that the password you choose here can't be altered or retrieved. - -```bash -ssh-keygen -t rsa -C "$your_email" -``` - -Use the code below to show your public key. - -```bash -cat ~/.ssh/id_rsa.pub -``` - -Copy-paste the key to the 'My SSH Keys' section under the 'SSH' tab in your user profile. Please copy the complete key starting with `ssh-` and ending with your username and host. diff --git a/doc/system_hooks/system_hooks.md b/doc/system_hooks/system_hooks.md index 54e6e3a9e3f..f9b6d37d840 100644 --- a/doc/system_hooks/system_hooks.md +++ b/doc/system_hooks/system_hooks.md @@ -1,6 +1,6 @@ # System hooks -Your GitLab instance can perform HTTP POST requests on the following events: `project_create`, `project_destroy`, `user_add_to_team`, `user_remove_from_team`, `user_create`, `user_destroy`, `key_create` and `key_destroy`. +Your GitLab instance can perform HTTP POST requests on the following events: `project_create`, `project_destroy`, `user_add_to_team`, `user_remove_from_team`, `user_create`, `user_destroy`, `key_create`, `key_destroy`, `group_create`, `group_destroy`, `user_add_to_group` and `user_remove_from_group`. System hooks can be used, e.g. for logging or changing information in a LDAP server. @@ -15,8 +15,8 @@ System hooks can be used, e.g. for logging or changing information in a LDAP ser "name": "StoreCloud", "owner_email": "johnsmith@gmail.com", "owner_name": "John Smith", - "path": "stormcloud", - "path_with_namespace": "jsmith/stormcloud", + "path": "storecloud", + "path_with_namespace": "jsmith/storecloud", "project_id": 74, "project_visibility": "private", } @@ -50,6 +50,7 @@ System hooks can be used, e.g. for logging or changing information in a LDAP ser "project_path": "storecloud", "user_email": "johnsmith@gmail.com", "user_name": "John Smith", + "user_id": 41, "project_visibility": "private", } ``` @@ -66,6 +67,7 @@ System hooks can be used, e.g. for logging or changing information in a LDAP ser "project_path": "storecloud", "user_email": "johnsmith@gmail.com", "user_name": "John Smith", + "user_id": 41, "project_visibility": "private", } ``` @@ -117,3 +119,62 @@ System hooks can be used, e.g. for logging or changing information in a LDAP ser "id": 4 } ``` + +**Group created:** + +```json +{ + "created_at": "2012-07-21T07:30:54Z", + "event_name": "group_create", + "name": "StoreCloud", + "owner_email": "johnsmith@gmail.com", + "owner_name": "John Smith", + "path": "storecloud", + "group_id": 78 +} +``` + +**Group removed:** + +```json +{ + "created_at": "2012-07-21T07:30:54Z", + "event_name": "group_destroy", + "name": "StoreCloud", + "owner_email": "johnsmith@gmail.com", + "owner_name": "John Smith", + "path": "storecloud", + "group_id": 78 +} +``` + +**New Group Member:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "event_name": "user_add_to_group", + "group_access": "Master", + "group_id": 78, + "group_name": "StoreCloud", + "group_path": "storecloud", + "user_email": "johnsmith@gmail.com", + "user_name": "John Smith", + "user_id": 41 +} +``` +**Group Member Removed:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "event_name": "user_remove_from_group", + "group_access": "Master", + "group_id": 78, + "group_name": "StoreCloud", + "group_path": "storecloud", + "user_email": "johnsmith@gmail.com", + "user_name": "John Smith", + "user_id": 41 +} +``` diff --git a/doc/update/2.6-to-3.0.md b/doc/update/2.6-to-3.0.md index 6aabbe095dc..4827ef9501a 100644 --- a/doc/update/2.6-to-3.0.md +++ b/doc/update/2.6-to-3.0.md @@ -1,4 +1,5 @@ # From 2.6 to 3.0 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/2.6-to-3.0.md) for the most up to date instructions.* ## 1. Stop server & resque @@ -22,29 +23,29 @@ sudo -u gitlab bundle exec rake db:migrate RAILS_ENV=production # !!! Config should be replaced with a new one. Check it after replace cp config/gitlab.yml.example config/gitlab.yml -# update gitolite hooks +# update Gitolite hooks -# GITOLITE v2: +# Gitolite v2: sudo cp ./lib/hooks/post-receive /home/git/share/gitolite/hooks/common/post-receive sudo chown git:git /home/git/share/gitolite/hooks/common/post-receive -# GITOLITE v3: +# Gitolite v3: sudo cp ./lib/hooks/post-receive /home/git/.gitolite/hooks/common/post-receive sudo chown git:git /home/git/.gitolite/hooks/common/post-receive # set valid path to hooks in gitlab.yml in git_host section # like this git_host: - # gitolite 2 + # Gitolite 2 hooks_path: /home/git/share/gitolite/hooks - # gitolite 3 + # Gitolite 3 hooks_path: /home/git/.gitolite/hooks/ -# Make some changes to gitolite config +# Make some changes to Gitolite config # For more information visit https://github.com/gitlabhq/gitlabhq/pull/1719 -# gitolite v2 +# Gitolite v2 sudo -u git -H sed -i 's/\(GL_GITCONFIG_KEYS\s*=>*\s*\).\{2\}/\\1"\.\*"/g' /home/git/.gitolite.rc # gitlite v3 diff --git a/doc/update/2.9-to-3.0.md b/doc/update/2.9-to-3.0.md index 8af86b0dc98..f4a997a8c5e 100644 --- a/doc/update/2.9-to-3.0.md +++ b/doc/update/2.9-to-3.0.md @@ -1,4 +1,5 @@ # From 2.9 to 3.0 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/2.9-to-3.0.md) for the most up to date instructions.* ## 1. Stop server & resque diff --git a/doc/update/3.0-to-3.1.md b/doc/update/3.0-to-3.1.md index 3206df3499b..a30485c42f7 100644 --- a/doc/update/3.0-to-3.1.md +++ b/doc/update/3.0-to-3.1.md @@ -1,4 +1,5 @@ # From 3.0 to 3.1 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/3.0-to-3.1.md) for the most up to date instructions.* **IMPORTANT!** diff --git a/doc/update/3.1-to-4.0.md b/doc/update/3.1-to-4.0.md index 165f4e6a308..f1ef4df4744 100644 --- a/doc/update/3.1-to-4.0.md +++ b/doc/update/3.1-to-4.0.md @@ -1,4 +1,5 @@ # From 3.1 to 4.0 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/3.1-to-4.0.md) for the most up to date instructions.* ## Important changes diff --git a/doc/update/4.0-to-4.1.md b/doc/update/4.0-to-4.1.md index 4149ed6b08d..d89d5235917 100644 --- a/doc/update/4.0-to-4.1.md +++ b/doc/update/4.0-to-4.1.md @@ -1,4 +1,5 @@ # From 4.0 to 4.1 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/4.0-to-4.1.md) for the most up to date instructions.* ## Important changes diff --git a/doc/update/4.1-to-4.2.md b/doc/update/4.1-to-4.2.md index 5ee8e8781e9..6fe4412ff90 100644 --- a/doc/update/4.1-to-4.2.md +++ b/doc/update/4.1-to-4.2.md @@ -1,4 +1,5 @@ # From 4.1 to 4.2 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/4.1-to-4.2.md) for the most up to date instructions.* ## 1. Stop server & Resque diff --git a/doc/update/4.2-to-5.0.md b/doc/update/4.2-to-5.0.md index 897cd0b91fa..f9faf65f952 100644 --- a/doc/update/4.2-to-5.0.md +++ b/doc/update/4.2-to-5.0.md @@ -1,4 +1,5 @@ # From 4.2 to 5.0 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/4.2-to-5.0.md) for the most up to date instructions.* ## Warning @@ -41,8 +42,8 @@ git checkout v1.1.0 # copy config cp config.yml.example config.yml -# change url to GitLab instance -# ! make sure url end with '/' like 'https://gitlab.example/' +# change URL to GitLab instance +# ! make sure the URL ends with '/' like 'https://gitlab.example/' vim config.yml # rewrite hooks @@ -111,7 +112,7 @@ sudo chmod -R u+rwX /home/git/gitlab/tmp/pids ``` -## 6. Update init.d script and nginx config +## 6. Update init.d script and Nginx config ```bash # init.d @@ -123,7 +124,7 @@ sudo chmod +x /etc/init.d/gitlab sudo -u git -H cp /home/git/gitlab/config/unicorn.rb /home/git/gitlab/config/unicorn.rb.old sudo -u git -H cp /home/git/gitlab/config/unicorn.rb.example /home/git/gitlab/config/unicorn.rb -#nginx +# Nginx # Replace path from '/home/gitlab/' to '/home/git/' sudo vim /etc/nginx/sites-enabled/gitlab sudo service nginx restart @@ -137,7 +138,7 @@ sudo service gitlab start # check if unicorn and sidekiq started # If not try to logout, also check replaced path from '/home/gitlab/' to '/home/git/' -# in nginx, unicorn, init.d etc +# in Nginx, unicorn, init.d etc ps aux | grep unicorn ps aux | grep sidekiq @@ -195,6 +196,12 @@ sudo rm -R tmp sudo -u git -H mkdir tmp sudo chmod -R u+rwX tmp/ +# create directory for pids, make sure GitLab can write to it +sudo -u git -H mkdir tmp/pids/ +sudo chmod -R u+rwX tmp/pids/ + +# if you are already running a newer version of GitLab check that installation guide for other tmp folders you need to create + # reboot system sudo reboot diff --git a/doc/update/5.0-to-5.1.md b/doc/update/5.0-to-5.1.md index 0e597abb1a9..9fbd1f88515 100644 --- a/doc/update/5.0-to-5.1.md +++ b/doc/update/5.0-to-5.1.md @@ -1,4 +1,5 @@ # From 5.0 to 5.1 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/5.0-to-5.1.md) for the most up to date instructions.* ## Warning diff --git a/doc/update/5.1-to-5.2.md b/doc/update/5.1-to-5.2.md index 6ef559ac9f9..cf9c4e4f770 100644 --- a/doc/update/5.1-to-5.2.md +++ b/doc/update/5.1-to-5.2.md @@ -1,4 +1,5 @@ # From 5.1 to 5.2 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/5.1-to-5.2.md) for the most up to date instructions.* ## Warning diff --git a/doc/update/5.1-to-5.4.md b/doc/update/5.1-to-5.4.md index 8ec56b266ca..97a98ede070 100644 --- a/doc/update/5.1-to-5.4.md +++ b/doc/update/5.1-to-5.4.md @@ -1,4 +1,5 @@ # From 5.1 to 5.4 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/5.1-to-5.4.md) for the most up to date instructions.* Also works starting from 5.2. diff --git a/doc/update/5.1-to-6.0.md b/doc/update/5.1-to-6.0.md index a76b371e6d6..a3fdd92bd2f 100644 --- a/doc/update/5.1-to-6.0.md +++ b/doc/update/5.1-to-6.0.md @@ -1,4 +1,5 @@ # From 5.1 to 6.0 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/5.1-to-6.0.md) for the most up to date instructions.* ## Warning @@ -40,7 +41,7 @@ sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production The migrations in this update are very sensitive to incomplete or inconsistent data. If you have a long-running GitLab installation and some of the previous upgrades did not work out 100% correct this may bite you now. The following can help you have a more smooth upgrade. -### Find projets with invalid project names +### Find projects with invalid project names #### MySQL Login to MySQL: diff --git a/doc/update/5.2-to-5.3.md b/doc/update/5.2-to-5.3.md index 61ddf135641..27613aeda07 100644 --- a/doc/update/5.2-to-5.3.md +++ b/doc/update/5.2-to-5.3.md @@ -1,4 +1,5 @@ # From 5.2 to 5.3 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/5.2-to-5.3.md) for the most up to date instructions.* ## Warning diff --git a/doc/update/5.3-to-5.4.md b/doc/update/5.3-to-5.4.md index 8a0d43e3e64..577b9a585ff 100644 --- a/doc/update/5.3-to-5.4.md +++ b/doc/update/5.3-to-5.4.md @@ -1,4 +1,5 @@ # From 5.3 to 5.4 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/5.3-to-5.4.md) for the most up to date instructions.* ## 0. Backup diff --git a/doc/update/5.4-to-6.0.md b/doc/update/5.4-to-6.0.md index 7bf7bce6aa0..d9c6d9bfb91 100644 --- a/doc/update/5.4-to-6.0.md +++ b/doc/update/5.4-to-6.0.md @@ -1,16 +1,20 @@ # From 5.4 to 6.0 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/5.4-to-6.0.md) for the most up to date instructions.* ## Warning GitLab 6.0 is affected by critical security vulnerabilities CVE-2013-4490 and CVE-2013-4489. +**You need to follow this guide first, before updating past 6.0, as it contains critical migration steps that are only present +in the `6-0-stable` branch** + ## Deprecations ### Global projects The root (global) namespace for projects is deprecated. -So you need to move all your global projects under groups or users manually before update or they will be automatically moved to the project owner namespace during the update. When a project is moved all its members will receive an email with instructions how to update their git remote url. Please make sure you disable sending email when you do a test of the upgrade. +So you need to move all your global projects under groups or users manually before update or they will be automatically moved to the project owner namespace during the update. When a project is moved all its members will receive an email with instructions how to update their git remote URL. Please make sure you disable sending email when you do a test of the upgrade. ### Teams diff --git a/doc/update/6.0-to-6.1.md b/doc/update/6.0-to-6.1.md index 9d67a3bcb96..c5eba1c01c4 100644 --- a/doc/update/6.0-to-6.1.md +++ b/doc/update/6.0-to-6.1.md @@ -1,4 +1,5 @@ # From 6.0 to 6.1 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.0-to-6.1.md) for the most up to date instructions.* ## Warning diff --git a/doc/update/6.1-to-6.2.md b/doc/update/6.1-to-6.2.md index efa6e43124c..a534528108a 100644 --- a/doc/update/6.1-to-6.2.md +++ b/doc/update/6.1-to-6.2.md @@ -1,4 +1,5 @@ # From 6.1 to 6.2 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.1-to-6.2.md) for the most up to date instructions.* **You should update to 6.1 before installing 6.2 so all the necessary conversions are run.** @@ -35,7 +36,7 @@ sudo -u git -H git checkout v1.7.9 # Addresses multiple critical security vulner ## 4. Install additional packages ```bash -# Add support for lograte for better log file handling +# Add support for logrotate for better log file handling sudo apt-get install logrotate ``` diff --git a/doc/update/6.2-to-6.3.md b/doc/update/6.2-to-6.3.md index e9b3bdd2f54..b08ebde0808 100644 --- a/doc/update/6.2-to-6.3.md +++ b/doc/update/6.2-to-6.3.md @@ -1,4 +1,5 @@ # From 6.2 to 6.3 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.2-to-6.3.md) for the most up to date instructions.* **Requires version: 6.1 or 6.2.** diff --git a/doc/update/6.3-to-6.4.md b/doc/update/6.3-to-6.4.md index 96c2895981d..951d92dfeb5 100644 --- a/doc/update/6.3-to-6.4.md +++ b/doc/update/6.3-to-6.4.md @@ -1,4 +1,5 @@ # From 6.3 to 6.4 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.3-to-6.4.md) for the most up to date instructions.* ## 0. Backup diff --git a/doc/update/6.4-to-6.5.md b/doc/update/6.4-to-6.5.md index 1624296fc3f..0dae9a9fe59 100644 --- a/doc/update/6.4-to-6.5.md +++ b/doc/update/6.4-to-6.5.md @@ -1,4 +1,5 @@ # From 6.4 to 6.5 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.4-to-6.5.md) for the most up to date instructions.* ## 0. Backup diff --git a/doc/update/6.5-to-6.6.md b/doc/update/6.5-to-6.6.md index 544eee17fec..c24e83eb006 100644 --- a/doc/update/6.5-to-6.6.md +++ b/doc/update/6.5-to-6.6.md @@ -1,4 +1,5 @@ # From 6.5 to 6.6 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.5-to-6.6.md) for the most up to date instructions.* ## 0. Backup diff --git a/doc/update/6.6-to-6.7.md b/doc/update/6.6-to-6.7.md index 77ac4d0bfa6..5622a7001ed 100644 --- a/doc/update/6.6-to-6.7.md +++ b/doc/update/6.6-to-6.7.md @@ -1,4 +1,5 @@ # From 6.6 to 6.7 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.6-to-6.7.md) for the most up to date instructions.* ## 0. Backup diff --git a/doc/update/6.7-to-6.8.md b/doc/update/6.7-to-6.8.md index 16f3439c998..4fb90639f16 100644 --- a/doc/update/6.7-to-6.8.md +++ b/doc/update/6.7-to-6.8.md @@ -1,4 +1,5 @@ # From 6.7 to 6.8 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.7-to-6.8.md) for the most up to date instructions.* ## 0. Backup diff --git a/doc/update/6.8-to-6.9.md b/doc/update/6.8-to-6.9.md index 9efb384ff59..b9b8b63f652 100644 --- a/doc/update/6.8-to-6.9.md +++ b/doc/update/6.8-to-6.9.md @@ -1,4 +1,5 @@ # From 6.8 to 6.9 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.8-to-6.9.md) for the most up to date instructions.* ### 0. Backup diff --git a/doc/update/6.9-to-7.0.md b/doc/update/6.9-to-7.0.md index 1f3421a799b..236430b5951 100644 --- a/doc/update/6.9-to-7.0.md +++ b/doc/update/6.9-to-7.0.md @@ -1,4 +1,5 @@ # From 6.9 to 7.0 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.9-to-7.0.md) for the most up to date instructions.* ### 0. Backup diff --git a/doc/update/6.x-or-7.x-to-7.3.md b/doc/update/6.x-or-7.x-to-7.9.md index 171fcb4033a..bd6eb6b211f 100644 --- a/doc/update/6.x-or-7.x-to-7.3.md +++ b/doc/update/6.x-or-7.x-to-7.9.md @@ -1,6 +1,7 @@ -# From 6.x or 7.x to 7.3 +# From 6.x or 7.x to 7.9 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/6.x-or-7.x-to-7.9.md) for the most up to date instructions.* -This allows you to upgrade any version of GitLab from 6.0 and up (including 7.0 and up) to 7.3. +This allows you to upgrade any version of GitLab from 6.0 and up (including 7.0 and up) to 7.9. ## Global issue numbers @@ -13,7 +14,11 @@ possible to edit the label text and color. The characters `?`, `&` and `,` are no longer allowed however so those will be removed from your tags during the database migrations for GitLab 7.2. -## 0. Backup +## 0. Stop server + + sudo service gitlab stop + +## 1. Backup It's useful to make a backup just in case things go south: (With MySQL, this may require granting "LOCK TABLES" privileges to the GitLab user on the database version) @@ -23,10 +28,6 @@ cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ``` -## 1. Stop server - - sudo service gitlab stop - ## 2. Update Ruby If you are still using Ruby 1.9.3 or below, you will need to update Ruby. @@ -34,7 +35,7 @@ You can check which version you are running with `ruby -v`. If you are you running Ruby 2.0.x, you do not need to upgrade ruby, but can consider doing so for performance reasons. -If you are running Ruby 2.1.1 consider upgrading to 2.1.2, because of the high memory usage of Ruby 2.1.1. +If you are running Ruby 2.1.1 consider upgrading to 2.1.5, because of the high memory usage of Ruby 2.1.1. Install, update dependencies: @@ -46,8 +47,8 @@ Download and compile Ruby: ```bash mkdir /tmp/ruby && cd /tmp/ruby -curl --progress ftp://ftp.ruby-lang.org/pub/ruby/2.1/ruby-2.1.2.tar.gz | tar xz -cd ruby-2.1.2 +curl --progress http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.5.tar.gz | tar xz +cd ruby-2.1.5 ./configure --disable-install-rdoc make sudo make install @@ -64,13 +65,13 @@ sudo gem install bundler --no-ri --no-rdoc ```bash cd /home/git/gitlab sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically ``` For GitLab Community Edition: ```bash -sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically -sudo -u git -H git checkout 7-3-stable +sudo -u git -H git checkout 7-9-stable ``` OR @@ -78,18 +79,23 @@ OR For GitLab Enterprise Edition: ```bash -sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically -sudo -u git -H git checkout 7-3-stable-ee +sudo -u git -H git checkout 7-9-stable-ee ``` ## 4. Install additional packages ```bash -# Add support for lograte for better log file handling +# Add support for logrotate for better log file handling sudo apt-get install logrotate # Install pkg-config and cmake, which is needed for the latest versions of rugged sudo apt-get install pkg-config cmake + +# Install Kerberos header files, which are needed for GitLab EE Kerberos support +sudo apt-get install libkrb5-dev + +# Install nodejs, javascript runtime required for assets +sudo apt-get install nodejs ``` ## 5. Configure Redis to use sockets @@ -100,6 +106,8 @@ sudo apt-get install pkg-config cmake sed 's/^port .*/port 0/' /etc/redis/redis.conf.orig | sudo tee /etc/redis/redis.conf # Enable Redis socket for default Debian / Ubuntu path echo 'unixsocket /var/run/redis/redis.sock' | sudo tee -a /etc/redis/redis.conf + # Be sure redis group can write to the socket, enable only if supported (>= redis 2.4.0). + sudo sed -i '/# unixsocketperm/ s/^# unixsocketperm.*/unixsocketperm 0775/' /etc/redis/redis.conf # Activate the changes to redis.conf sudo service redis-server restart # Add git to the redis group @@ -118,7 +126,7 @@ sudo apt-get install pkg-config cmake ```bash cd /home/git/gitlab-shell sudo -u git -H git fetch -sudo -u git -H git checkout v2.0.1 +sudo -u git -H git checkout v2.6.0 ``` ## 7. Install libs, migrations, etc. @@ -153,14 +161,12 @@ sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab TIP: to see what changed in `gitlab.yml.example` in this release use next command: ``` -git diff 6-0-stable:config/gitlab.yml.example 7-3-stable:config/gitlab.yml.example +git diff 6-0-stable:config/gitlab.yml.example 7-9-stable:config/gitlab.yml.example ``` -* Make `/home/git/gitlab/config/gitlab.yml` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-3-stable/config/gitlab.yml.example but with your settings. -* Make `/home/git/gitlab/config/unicorn.rb` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-3-stable/config/unicorn.rb.example but with your settings. -* Make `/home/git/gitlab-shell/config.yml` the same as https://gitlab.com/gitlab-org/gitlab-shell/blob/v2.0.0/config.yml.example but with your settings. -* HTTP setups: Make `/etc/nginx/sites-available/nginx` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-3-stable/lib/support/nginx/gitlab but with your settings. -* HTTPS setups: Make `/etc/nginx/sites-available/nginx-ssl` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-3-stable/lib/support/nginx/gitlab-ssl but with your settings. +* Make `/home/git/gitlab/config/gitlab.yml` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-9-stable/config/gitlab.yml.example but with your settings. +* Make `/home/git/gitlab/config/unicorn.rb` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-9-stable/config/unicorn.rb.example but with your settings. +* Make `/home/git/gitlab-shell/config.yml` the same as https://gitlab.com/gitlab-org/gitlab-shell/blob/v2.6.0/config.yml.example but with your settings. * Copy rack attack middleware config ```bash @@ -173,6 +179,12 @@ sudo -u git -H cp config/initializers/rack_attack.rb.example config/initializers sudo cp lib/support/logrotate/gitlab /etc/logrotate.d/gitlab ``` +### Change Nginx settings + +* HTTP setups: Make `/etc/nginx/sites-available/gitlab` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-9-stable/lib/support/nginx/gitlab but with your settings. +* HTTPS setups: Make `/etc/nginx/sites-available/gitlab-ssl` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-9-stable/lib/support/nginx/gitlab-ssl but with your settings. +* A new `location /uploads/` section has been added that needs to have the same content as the existing `location @gitlab` section. + ## 9. Start application sudo service gitlab start @@ -195,13 +207,84 @@ If all items are green, then congratulations upgrade complete! When using Google omniauth login, changes of the Google account required. Ensure that `Contacts API` and the `Google+ API` are enabled in the [Google Developers Console](https://console.developers.google.com/). -More details can be found at the [integration documentation](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/integration/google.md). +More details can be found at the [integration documentation](../../../master/doc/integration/google.md). + +## 12. Optional optimizations for GitLab setups with MySQL databases + +Only applies if running MySQL database created with GitLab 6.7 or earlier. If you are not experiencing any issues you may not need the following instructions however following them will bring your database in line with the latest recommended installation configuration and help avoid future issues. Be sure to follow these directions exactly. These directions should be safe for any MySQL instance but to be sure make a current MySQL database backup beforehand. + +``` +# Stop GitLab +sudo service gitlab stop + +# Secure your MySQL installation (added in GitLab 6.2) +sudo mysql_secure_installation + +# Login to MySQL +mysql -u root -p + +# do not type the 'mysql>', this is part of the prompt + +# Convert all tables to use the InnoDB storage engine (added in GitLab 6.8) +SELECT CONCAT('ALTER TABLE gitlabhq_production.', table_name, ' ENGINE=InnoDB;') AS 'Copy & run these SQL statements:' FROM information_schema.tables WHERE table_schema = 'gitlabhq_production' AND `ENGINE` <> 'InnoDB' AND `TABLE_TYPE` = 'BASE TABLE'; + +# If previous query returned results, copy & run all shown SQL statements + +# Convert all tables to correct character set +SET foreign_key_checks = 0; +SELECT CONCAT('ALTER TABLE gitlabhq_production.', table_name, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;') AS 'Copy & run these SQL statements:' FROM information_schema.tables WHERE table_schema = 'gitlabhq_production' AND `TABLE_COLLATION` <> 'utf8_unicode_ci' AND `TABLE_TYPE` = 'BASE TABLE'; + +# If previous query returned results, copy & run all shown SQL statements + +# turn foreign key checks back on +SET foreign_key_checks = 1; + +# Find MySQL users +mysql> SELECT user FROM mysql.user WHERE user LIKE '%git%'; + +# If git user exists and gitlab user does not exist +# you are done with the database cleanup tasks +mysql> \q + +# If both users exist skip to Delete gitlab user + +# Create new user for GitLab (changed in GitLab 6.4) +# change $password in the command below to a real password you pick +mysql> CREATE USER 'git'@'localhost' IDENTIFIED BY '$password'; + +# Grant the git user necessary permissions on the database +mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, LOCK TABLES ON `gitlabhq_production`.* TO 'git'@'localhost'; + +# Delete the old gitlab user +mysql> DELETE FROM mysql.user WHERE user='gitlab'; + +# Quit the database session +mysql> \q + +# Try connecting to the new database with the new user +sudo -u git -H mysql -u git -p -D gitlabhq_production + +# Type the password you replaced $password with earlier + +# You should now see a 'mysql>' prompt + +# Quit the database session +mysql> \q + +# Update database configuration details +# See config/database.yml.mysql for latest recommended configuration details +# Remove the reaping_frequency setting line if it exists (removed in GitLab 6.8) +# Set production -> pool: 10 (updated in GitLab 5.3) +# Set production -> username: git +# Set production -> password: the password your replaced $password with earlier +sudo -u git -H editor /home/git/gitlab/config/database.yml +``` -## Things went south? Revert to previous version (6.0) +## Things went south? Revert to previous version (7.0) ### 1. Revert the code to the previous version -Follow the [upgrade guide from 5.4 to 6.0](5.4-to-6.0.md), except for the database migration (the backup is already migrated to the previous version). +Follow the [upgrade guide from 6.9 to 7.0](6.9-to-7.0.md), except for the database migration (the backup is already migrated to the previous version). ### 2. Restore from the backup: diff --git a/doc/update/7.0-to-7.1.md b/doc/update/7.0-to-7.1.md index 82bb5708734..a4e9be9946e 100644 --- a/doc/update/7.0-to-7.1.md +++ b/doc/update/7.0-to-7.1.md @@ -1,4 +1,5 @@ # From 7.0 to 7.1 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/7.0-to-7.1.md) for the most up to date instructions.* ### 0. Backup diff --git a/doc/update/7.1-to-7.2.md b/doc/update/7.1-to-7.2.md index 699111f0143..88cb63d7d41 100644 --- a/doc/update/7.1-to-7.2.md +++ b/doc/update/7.1-to-7.2.md @@ -1,4 +1,5 @@ # From 7.1 to 7.2 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/7.1-to-7.2.md) for the most up to date instructions.* ## Editable labels diff --git a/doc/update/7.2-to-7.3.md b/doc/update/7.2-to-7.3.md index 329b763322a..18f77d6396e 100644 --- a/doc/update/7.2-to-7.3.md +++ b/doc/update/7.2-to-7.3.md @@ -1,4 +1,5 @@ # From 7.2 to 7.3 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/7.2-to-7.3.md) for the most up to date instructions.* ### 0. Backup @@ -18,12 +19,12 @@ sudo service gitlab stop ```bash cd /home/git/gitlab sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically ``` For GitLab Community Edition: ```bash -sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically sudo -u git -H git checkout 7-3-stable ``` @@ -32,7 +33,6 @@ OR For GitLab Enterprise Edition: ```bash -sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically sudo -u git -H git checkout 7-3-stable-ee ``` @@ -75,7 +75,7 @@ sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab # Enable Redis socket for default Debian / Ubuntu path echo 'unixsocket /var/run/redis/redis.sock' | sudo tee -a /etc/redis/redis.conf # Be sure redis group can write to the socket, enable only if supported (>= redis 2.4.0). - sed -i '/# unixsocketperm/ s/^# unixsocketperm.*/unixsocketperm 0775/' /etc/redis/redis.conf + sudo sed -i '/# unixsocketperm/ s/^# unixsocketperm.*/unixsocketperm 0775/' /etc/redis/redis.conf # Activate the changes to redis.conf sudo service redis-server restart # Add git to the redis group diff --git a/doc/update/7.3-to-7.4.md b/doc/update/7.3-to-7.4.md new file mode 100644 index 00000000000..53e739c06fb --- /dev/null +++ b/doc/update/7.3-to-7.4.md @@ -0,0 +1,197 @@ +# From 7.3 to 7.4 +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/7.3-to-7.4.md) for the most up to date instructions.* + +### 0. Stop server + + sudo service gitlab stop + +### 1. Backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 2. Get latest code + +```bash +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +``` + +For GitLab Community Edition: + +```bash +sudo -u git -H git checkout 7-4-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +sudo -u git -H git checkout 7-4-stable-ee +``` + +### 3. Install libs, migrations, etc. + +```bash +# MySQL installations (note: the line below states '--without ... postgres') +sudo -u git -H bundle install --without development test postgres --deployment + +# PostgreSQL installations (note: the line below states '--without ... mysql') +sudo -u git -H bundle install --without development test mysql --deployment + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Clean up assets and cache +sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production + +# Update init.d script +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +### 4. Update config files + +#### New configuration options for gitlab.yml + +There are new configuration options available for gitlab.yml. View them with the command below and apply them to your current gitlab.yml. + +``` +git diff origin/7-3-stable:config/gitlab.yml.example origin/7-4-stable:config/gitlab.yml.example +``` + +#### Change timeout for unicorn + +``` +# set timeout to 60 +sudo -u git -H editor config/unicorn.rb +``` + +#### Change Nginx HTTPS settings + +* HTTPS setups: Make `/etc/nginx/sites-available/gitlab-ssl` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/7-4-stable/lib/support/nginx/gitlab-ssl but with your setting + +#### MySQL Databases: Update database.yml config file + +* Add `collation: utf8_general_ci` to `config/database.yml` as seen in [config/database.yml.mysql](/config/database.yml.mysql) + +``` +sudo -u git -H editor config/database.yml +``` + +### 5. Start application + + sudo service gitlab start + sudo service nginx restart + +### 6. Check application status + +Check if GitLab and its environment are configured correctly: + + sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production + +To make sure you didn't miss anything run a more thorough check with: + + sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production + +If all items are green, then congratulations upgrade is complete! + + +### 7. Optional optimizations for GitLab setups with MySQL databases + +Only applies if running MySQL database created with GitLab 6.7 or earlier. If you are not experiencing any issues you may not need the following instructions however following them will bring your database in line with the latest recommended installation configuration and help avoid future issues. Be sure to follow these directions exactly. These directions should be safe for any MySQL instance but to be sure make a current MySQL database backup beforehand. + +``` +# Stop GitLab +sudo service gitlab stop + +# Secure your MySQL installation (added in GitLab 6.2) +sudo mysql_secure_installation + +# Login to MySQL +mysql -u root -p + +# do not type the 'mysql>', this is part of the prompt + +# Convert all tables to use the InnoDB storage engine (added in GitLab 6.8) +SELECT CONCAT('ALTER TABLE gitlabhq_production.', table_name, ' ENGINE=InnoDB;') AS 'Copy & run these SQL statements:' FROM information_schema.tables WHERE table_schema = 'gitlabhq_production' AND `ENGINE` <> 'InnoDB' AND `TABLE_TYPE` = 'BASE TABLE'; + +# If previous query returned results, copy & run all shown SQL statements + +# Convert all tables to correct character set +SET foreign_key_checks = 0; +SELECT CONCAT('ALTER TABLE gitlabhq_production.', table_name, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;') AS 'Copy & run these SQL statements:' FROM information_schema.tables WHERE table_schema = 'gitlabhq_production' AND `TABLE_COLLATION` <> 'utf8_unicode_ci' AND `TABLE_TYPE` = 'BASE TABLE'; + +# If previous query returned results, copy & run all shown SQL statements + +# turn foreign key checks back on +SET foreign_key_checks = 1; + +# Find MySQL users +mysql> SELECT user FROM mysql.user WHERE user LIKE '%git%'; + +# If git user exists and gitlab user does not exist +# you are done with the database cleanup tasks +mysql> \q + +# If both users exist skip to Delete gitlab user + +# Create new user for GitLab (changed in GitLab 6.4) +# change $password in the command below to a real password you pick +mysql> CREATE USER 'git'@'localhost' IDENTIFIED BY '$password'; + +# Grant the git user necessary permissions on the database +mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, LOCK TABLES ON `gitlabhq_production`.* TO 'git'@'localhost'; + +# Delete the old gitlab user +mysql> DELETE FROM mysql.user WHERE user='gitlab'; + +# Quit the database session +mysql> \q + +# Try connecting to the new database with the new user +sudo -u git -H mysql -u git -p -D gitlabhq_production + +# Type the password you replaced $password with earlier + +# You should now see a 'mysql>' prompt + +# Quit the database session +mysql> \q + +# Update database configuration details +# See config/database.yml.mysql for latest recommended configuration details +# Remove the reaping_frequency setting line if it exists (removed in GitLab 6.8) +# Set production -> pool: 10 (updated in GitLab 5.3) +# Set production -> username: git +# Set production -> password: the password your replaced $password with earlier +sudo -u git -H editor /home/git/gitlab/config/database.yml + +# Start GitLab +sudo service gitlab start +sudo service nginx restart + +# Run thorough check +sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production +``` + + +## Things went south? Revert to previous version (7.3) + +### 1. Revert the code to the previous version +Follow the [upgrade guide from 7.2 to 7.3](7.2-to-7.3.md), except for the database migration +(The backup is already migrated to the previous version) + +### 2. Restore from the backup: + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` +If you have more than one backup *.tar file(s) please add `BACKUP=timestamp_of_backup` to the command above. + + + + diff --git a/doc/update/7.4-to-7.5.md b/doc/update/7.4-to-7.5.md new file mode 100644 index 00000000000..673eab3c56e --- /dev/null +++ b/doc/update/7.4-to-7.5.md @@ -0,0 +1,108 @@ +# From 7.4 to 7.5 + +### 0. Stop server + + sudo service gitlab stop + +### 1. Backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 2. Get latest code + +```bash +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +``` + +For GitLab Community Edition: + +```bash +sudo -u git -H git checkout 7-5-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +sudo -u git -H git checkout 7-5-stable-ee +``` + +### 3. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell +sudo -u git -H git fetch +sudo -u git -H git checkout v2.2.0 +``` + +### 4. Install libs, migrations, etc. + +```bash +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without ... postgres') +sudo -u git -H bundle install --without development test postgres --deployment + +# PostgreSQL installations (note: the line below states '--without ... mysql') +sudo -u git -H bundle install --without development test mysql --deployment + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Clean up assets and cache +sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production + +# Update init.d script +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +### 5. Update config files + +#### New configuration options for gitlab.yml + +There are new configuration options available for gitlab.yml. View them with the command below and apply them to your current gitlab.yml. + +``` +git diff origin/7-4-stable:config/gitlab.yml.example origin/7-5-stable:config/gitlab.yml.example +``` + +#### Change Nginx settings + +* HTTP setups: Make `/etc/nginx/sites-available/gitlab` the same as [`lib/support/nginx/gitlab`](/lib/support/nginx/gitlab) but with your settings +* HTTPS setups: Make `/etc/nginx/sites-available/gitlab-ssl` the same as [`lib/support/nginx/gitlab-ssl`](/lib/support/nginx/gitlab-ssl) but with your setting + +### 6. Start application + + sudo service gitlab start + sudo service nginx restart + +### 7. Check application status + +Check if GitLab and its environment are configured correctly: + + sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production + +To make sure you didn't miss anything run a more thorough check with: + + sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production + +If all items are green, then congratulations upgrade is complete! + +## Things went south? Revert to previous version (7.4) + +### 1. Revert the code to the previous version +Follow the [upgrade guide from 7.3 to 7.4](7.3-to-7.4.md), except for the database migration +(The backup is already migrated to the previous version) + +### 2. Restore from the backup: + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` +If you have more than one backup *.tar file(s) please add `BACKUP=timestamp_of_backup` to the command above. diff --git a/doc/update/7.5-to-7.6.md b/doc/update/7.5-to-7.6.md new file mode 100644 index 00000000000..35cd437fdc4 --- /dev/null +++ b/doc/update/7.5-to-7.6.md @@ -0,0 +1,114 @@ +# From 7.5 to 7.6 + +### 0. Stop server + + sudo service gitlab stop + +### 1. Backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 2. Get latest code + +```bash +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +``` + +For GitLab Community Edition: + +```bash +sudo -u git -H git checkout 7-6-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +sudo -u git -H git checkout 7-6-stable-ee +``` + +### 3. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell +sudo -u git -H git fetch +sudo -u git -H git checkout v2.4.0 +``` + +### 4. Install libs, migrations, etc. + +```bash +sudo apt-get install libkrb5-dev + +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without ... postgres') +sudo -u git -H bundle install --without development test postgres --deployment + +# PostgreSQL installations (note: the line below states '--without ... mysql') +sudo -u git -H bundle install --without development test mysql --deployment + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Clean up assets and cache +sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production + +# Update init.d script +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +### 5. Update config files + +#### New configuration options for `gitlab.yml` + +There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). View them with the command below and apply them to your current `gitlab.yml`. + +``` +git diff origin/7-5-stable:config/gitlab.yml.example origin/7-6-stable:config/gitlab.yml.example +``` + +#### Change Nginx settings + +* HTTP setups: Make `/etc/nginx/sites-available/gitlab` the same as [`lib/support/nginx/gitlab`](/lib/support/nginx/gitlab) but with your settings +* HTTPS setups: Make `/etc/nginx/sites-available/gitlab-ssl` the same as [`lib/support/nginx/gitlab-ssl`](/lib/support/nginx/gitlab-ssl) but with your setting + +#### Setup time zone (optional) + +Consider setting the time zone in `gitlab.yml` otherwise GitLab will default to UTC. If you set a time zone previously in [`application.rb`](config/application.rb) (unlikely), unset it. + +### 6. Start application + + sudo service gitlab start + sudo service nginx restart + +### 7. Check application status + +Check if GitLab and its environment are configured correctly: + + sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production + +To make sure you didn't miss anything run a more thorough check with: + + sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production + +If all items are green, then congratulations upgrade is complete! + +## Things went south? Revert to previous version (7.5) + +### 1. Revert the code to the previous version +Follow the [upgrade guide from 7.4 to 7.5](7.4-to-7.5.md), except for the database migration +(The backup is already migrated to the previous version) + +### 2. Restore from the backup: + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` +If you have more than one backup *.tar file(s) please add `BACKUP=timestamp_of_backup` to the command above. diff --git a/doc/update/7.6-to-7.7.md b/doc/update/7.6-to-7.7.md new file mode 100644 index 00000000000..59243713156 --- /dev/null +++ b/doc/update/7.6-to-7.7.md @@ -0,0 +1,119 @@ +# From 7.6 to 7.7 + +### 0. Stop server + + sudo service gitlab stop + +### 1. Backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 2. Get latest code + +```bash +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +``` + +For GitLab Community Edition: + +```bash +sudo -u git -H git checkout 7-7-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +sudo -u git -H git checkout 7-7-stable-ee +``` + +### 3. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell +sudo -u git -H git fetch +sudo -u git -H git checkout v2.4.2 +``` + +### 4. Install libs, migrations, etc. + +```bash +sudo apt-get install libkrb5-dev + +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without ... postgres') +sudo -u git -H bundle install --without development test postgres --deployment + +# PostgreSQL installations (note: the line below states '--without ... mysql') +sudo -u git -H bundle install --without development test mysql --deployment + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Clean up assets and cache +sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production + +# Update init.d script +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +### 5. Update config files + +#### New configuration options for `gitlab.yml` + +There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). View them with the command below and apply them to your current `gitlab.yml`. + +``` +git diff origin/7-6-stable:config/gitlab.yml.example origin/7-7-stable:config/gitlab.yml.example +``` + +#### Change Nginx settings + +* HTTP setups: Make `/etc/nginx/sites-available/gitlab` the same as [`lib/support/nginx/gitlab`](/lib/support/nginx/gitlab) but with your settings +* HTTPS setups: Make `/etc/nginx/sites-available/gitlab-ssl` the same as [`lib/support/nginx/gitlab-ssl`](/lib/support/nginx/gitlab-ssl) but with your setting + +#### Setup time zone (optional) + +Consider setting the time zone in `gitlab.yml` otherwise GitLab will default to UTC. If you set a time zone previously in [`application.rb`](config/application.rb) (unlikely), unset it. + +### 6. Start application + + sudo service gitlab start + sudo service nginx restart + +### 7. Check application status + +Check if GitLab and its environment are configured correctly: + + sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production + +To make sure you didn't miss anything run a more thorough check with: + + sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production + +If all items are green, then congratulations upgrade is complete! + +### 8. GitHub settings (if applicable) + +If you are using GitHub as an OAuth provider for authentication, you should change the callback URL so that it +only contains a root URL (ex. `https://gitlab.example.com/`) + +## Things went south? Revert to previous version (7.6) + +### 1. Revert the code to the previous version +Follow the [upgrade guide from 7.5 to 7.6](7.5-to-7.6.md), except for the database migration +(The backup is already migrated to the previous version) + +### 2. Restore from the backup: + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` +If you have more than one backup *.tar file(s) please add `BACKUP=timestamp_of_backup` to the command above. diff --git a/doc/update/7.7-to-7.8.md b/doc/update/7.7-to-7.8.md new file mode 100644 index 00000000000..46ca163c1bb --- /dev/null +++ b/doc/update/7.7-to-7.8.md @@ -0,0 +1,120 @@ +# From 7.7 to 7.8 + +### 0. Stop server + + sudo service gitlab stop + +### 1. Backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 2. Get latest code + +```bash +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +``` + +For GitLab Community Edition: + +```bash +sudo -u git -H git checkout 7-8-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +sudo -u git -H git checkout 7-8-stable-ee +``` + +### 3. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell +sudo -u git -H git fetch +sudo -u git -H git checkout v2.5.4 +``` + +### 4. Install libs, migrations, etc. + +```bash +sudo apt-get install libkrb5-dev + +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without ... postgres') +sudo -u git -H bundle install --without development test postgres --deployment + +# PostgreSQL installations (note: the line below states '--without ... mysql') +sudo -u git -H bundle install --without development test mysql --deployment + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Clean up assets and cache +sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production + +# Update init.d script +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +### 5. Update config files + +#### New configuration options for `gitlab.yml` + +There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). View them with the command below and apply them to your current `gitlab.yml`. + +``` +git diff origin/7-7-stable:config/gitlab.yml.example origin/7-8-stable:config/gitlab.yml.example +``` + +#### Change Nginx settings + +* HTTP setups: Make `/etc/nginx/sites-available/gitlab` the same as [`lib/support/nginx/gitlab`](/lib/support/nginx/gitlab) but with your settings. +* HTTPS setups: Make `/etc/nginx/sites-available/gitlab-ssl` the same as [`lib/support/nginx/gitlab-ssl`](/lib/support/nginx/gitlab-ssl) but with your settings. +* A new `location /uploads/` section has been added that needs to have the same content as the existing `location @gitlab` section. + +#### Setup time zone (optional) + +Consider setting the time zone in `gitlab.yml` otherwise GitLab will default to UTC. If you set a time zone previously in [`application.rb`](config/application.rb) (unlikely), unset it. + +### 6. Start application + + sudo service gitlab start + sudo service nginx restart + +### 7. Check application status + +Check if GitLab and its environment are configured correctly: + + sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production + +To make sure you didn't miss anything run a more thorough check with: + + sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production + +If all items are green, then congratulations upgrade is complete! + +### 8. GitHub settings (if applicable) + +If you are using GitHub as an OAuth provider for authentication, you should change the callback URL so that it +only contains a root URL (ex. `https://gitlab.example.com/`) + +## Things went south? Revert to previous version (7.7) + +### 1. Revert the code to the previous version +Follow the [upgrade guide from 7.6 to 7.7](7.6-to-7.7.md), except for the database migration +(The backup is already migrated to the previous version) + +### 2. Restore from the backup: + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` +If you have more than one backup *.tar file(s) please add `BACKUP=timestamp_of_backup` to the command above. diff --git a/doc/update/7.8-to-7.9.md b/doc/update/7.8-to-7.9.md new file mode 100644 index 00000000000..28fd433e1c8 --- /dev/null +++ b/doc/update/7.8-to-7.9.md @@ -0,0 +1,120 @@ +# From 7.8 to 7.9 + +### 0. Stop server + + sudo service gitlab stop + +### 1. Backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 2. Get latest code + +```bash +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +``` + +For GitLab Community Edition: + +```bash +sudo -u git -H git checkout 7-9-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +sudo -u git -H git checkout 7-9-stable-ee +``` + +### 3. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell +sudo -u git -H git fetch +sudo -u git -H git checkout v2.6.0 +``` + +### 4. Install libs, migrations, etc. + +```bash +sudo apt-get install nodejs + +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without ... postgres') +sudo -u git -H bundle install --without development test postgres --deployment + +# PostgreSQL installations (note: the line below states '--without ... mysql') +sudo -u git -H bundle install --without development test mysql --deployment + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Clean up assets and cache +sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production + +# Update init.d script +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +### 5. Update config files + +#### New configuration options for `gitlab.yml` + +There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). View them with the command below and apply them to your current `gitlab.yml`. + +``` +git diff origin/7-8-stable:config/gitlab.yml.example origin/7-9-stable:config/gitlab.yml.example +``` + +#### Change Nginx settings + +* HTTP setups: Make `/etc/nginx/sites-available/gitlab` the same as [`lib/support/nginx/gitlab`](/lib/support/nginx/gitlab) but with your settings. +* HTTPS setups: Make `/etc/nginx/sites-available/gitlab-ssl` the same as [`lib/support/nginx/gitlab-ssl`](/lib/support/nginx/gitlab-ssl) but with your settings. +* A new `location /uploads/` section has been added that needs to have the same content as the existing `location @gitlab` section. + +#### Setup time zone (optional) + +Consider setting the time zone in `gitlab.yml` otherwise GitLab will default to UTC. If you set a time zone previously in [`application.rb`](config/application.rb) (unlikely), unset it. + +### 6. Start application + + sudo service gitlab start + sudo service nginx restart + +### 7. Check application status + +Check if GitLab and its environment are configured correctly: + + sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production + +To make sure you didn't miss anything run a more thorough check with: + + sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production + +If all items are green, then congratulations upgrade is complete! + +### 8. GitHub settings (if applicable) + +If you are using GitHub as an OAuth provider for authentication, you should change the callback URL so that it +only contains a root URL (ex. `https://gitlab.example.com/`) + +## Things went south? Revert to previous version (7.8) + +### 1. Revert the code to the previous version +Follow the [upgrade guide from 7.7 to 7.8](7.7-to-7.8.md), except for the database migration +(The backup is already migrated to the previous version) + +### 2. Restore from the backup: + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` +If you have more than one backup *.tar file(s) please add `BACKUP=timestamp_of_backup` to the command above. diff --git a/doc/update/README.md b/doc/update/README.md index 30e9137d7b7..0472537eeb5 100644 --- a/doc/update/README.md +++ b/doc/update/README.md @@ -4,13 +4,13 @@ Depending on the installation method and your GitLab version, there are multiple - [Omnibus update guide](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/update.md) contains the steps needed to update a GitLab [package](https://about.gitlab.com/downloads/). -## Manual Installation +## Installation from source -- [The individual upgrade guides](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update) are for those who have installed GitLab manually. +- [The individual upgrade guides](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update) are for those who have installed GitLab from source. - [The CE to EE update guides](https://gitlab.com/subscribers/gitlab-ee/tree/master/doc/update) are for subscribers of the Enterprise Edition only. The steps are very similar to a version upgrade: stop the server, get the code, update config files for the new functionality, install libs and do migrations, update the init script, start the application and check the application status. -- [Upgrader](upgrader.md) is an automatic ruby script that performs the update for manual installations. +- [Upgrader](upgrader.md) is an automatic ruby script that performs the update for installations from source. - [Patch versions](patch_versions.md) guide includes the steps needed for a patch version, eg. 6.2.0 to 6.2.1. ## Miscellaneous -- [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating your database from MySQL to PostrgreSQL. +- [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating your database from MySQL to PostgreSQL. diff --git a/doc/update/mysql_to_postgresql.md b/doc/update/mysql_to_postgresql.md index 695c083d361..50941db25f6 100644 --- a/doc/update/mysql_to_postgresql.md +++ b/doc/update/mysql_to_postgresql.md @@ -1,4 +1,5 @@ # Migrating GitLab from MySQL to Postgres +*Make sure you view this [guide from the `master` branch](../../../master/doc/update/mysql_to_postgresql.md) for the most up to date instructions.* If you are replacing MySQL with Postgres while keeping GitLab on the same server all you need to do is to export from MySQL, import into Postgres and rebuild the indexes as described below. If you are also moving GitLab to another server, or if you are switching to omnibus-gitlab, you may want to use a GitLab backup file. The second part of this documents explains the procedure to do this. @@ -11,9 +12,9 @@ sudo service gitlab stop # Update /home/git/gitlab/config/database.yml -git clone https://github.com/gitlabhq/mysql-postgresql-converter.git +git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab cd mysql-postgresql-converter -mysqldump --compatible=postgresql --default-character-set=utf8 -r databasename.mysql -u root gitlabhq_production +mysqldump --compatible=postgresql --default-character-set=utf8 -r databasename.mysql -u root gitlabhq_production -p python db_converter.py databasename.mysql databasename.psql # Import the database dump as the application database user @@ -38,7 +39,7 @@ On non-omnibus installations (distributed using Git) we retrieve the index decla ``` # Clone the database converter on your Postgres-backed GitLab server cd /tmp -git clone https://github.com/gitlabhq/mysql-postgresql-converter.git +git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab cd /home/git/gitlab @@ -59,7 +60,7 @@ On omnibus-gitlab we need to get the index declarations from a file called `sche ``` # Clone the database converter on your Postgres-backed GitLab server cd /tmp -/opt/gitlab/embedded/bin/git clone https://github.com/gitlabhq/mysql-postgresql-converter.git +/opt/gitlab/embedded/bin/git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab cd /tmp/mysql-postgresql-converter # Download schema.rb.bundled if necessary @@ -75,7 +76,7 @@ test -e /opt/gitlab/embedded/service/gitlab-rails/db/schema.rb.bundled || sudo / ## Converting a GitLab backup file from MySQL to Postgres **Note:** Please make sure to have Python 2.7.x (or higher) installed. -GitLab backup files (<timestamp>_gitlab_backup.tar) contain a SQL dump. Using the lanyrd database converter we can replace a MySQL database dump inside the tar file with a Postgres database dump. This can be useful if you are moving to another server. +GitLab backup files (`<timestamp>_gitlab_backup.tar`) contain a SQL dump. Using the lanyrd database converter we can replace a MySQL database dump inside the tar file with a Postgres database dump. This can be useful if you are moving to another server. ``` # Stop GitLab @@ -94,10 +95,10 @@ sudo -u git -H mv tmp/backups/TIMESTAMP_gitlab_backup.tar tmp/backups/postgresql # Create a separate database dump with PostgreSQL compatibility cd tmp/backups/postgresql -sudo -u git -H mysqldump --compatible=postgresql --default-character-set=utf8 -r gitlabhq_production.mysql -u root gitlabhq_production +sudo -u git -H mysqldump --compatible=postgresql --default-character-set=utf8 -r gitlabhq_production.mysql -u root gitlabhq_production -p # Clone the database converter -sudo -u git -H git clone https://github.com/gitlabhq/mysql-postgresql-converter.git +sudo -u git -H git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab # Convert gitlabhq_production.mysql sudo -u git -H mkdir db diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index c4a77d12800..ad302492556 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -1,4 +1,5 @@ # Universal update guide for patch versions +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/patch_versions.md) for the most up to date instructions.* For example from 6.2.0 to 6.2.1, also see the [semantic versioning specification](http://semver.org/). @@ -26,16 +27,14 @@ sudo -u git -H git checkout LATEST_TAG Replace LATEST_TAG with the latest GitLab tag you want to upgrade to, for example `v6.6.3`. -### 3. Update gitlab-shell if it is not the latest version +### 3. Update gitlab-shell to the corresponding version ```bash cd /home/git/gitlab-shell sudo -u git -H git fetch -sudo -u git -H git checkout LATEST_TAG +sudo -u git -H git checkout v`cat /home/git/gitlab/GITLAB_SHELL_VERSION` ``` -Replace LATEST_TAG with the latest GitLab Shell tag you want to upgrade to, for example `v1.7.9`. - ### 4. Install libs, migrations, etc. ```bash diff --git a/doc/update/upgrader.md b/doc/update/upgrader.md index cf59b0e461c..d8476fb3457 100644 --- a/doc/update/upgrader.md +++ b/doc/update/upgrader.md @@ -1,4 +1,5 @@ # GitLab Upgrader +*Make sure you view this [upgrade guide from the `master` branch](../../../master/doc/update/upgrader.md) for the most up to date instructions.* GitLab Upgrader - a ruby script that allows you easily upgrade GitLab to latest minor version. @@ -10,6 +11,8 @@ If you have local changes to your GitLab repository the script will stash them a **GitLab Upgrader is available only for GitLab version 6.4.2 or higher.** +**This script does NOT update gitlab-shell, it needs manual update. See step 5 below.** + ## 0. Backup cd /home/git/gitlab @@ -21,7 +24,7 @@ If you have local changes to your GitLab repository the script will stash them a ## 2. Run GitLab upgrade tool -Note: GitLab 7.2 adds `pkg-config` and `cmake` as dependency. Please check the dependencies in the [installation guide.](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md#1-packages-dependencies) +Note: GitLab 7.9 adds nodejs as a dependency. GitLab 7.6 adds `libkrb5-dev` as a dependency (installed by default on Ubuntu and OSX). GitLab 7.2 adds `pkg-config` and `cmake` as dependency. Please check the dependencies in the [installation guide.](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md#1-packages-dependencies) # Starting with GitLab version 7.0 upgrader script has been moved to bin directory cd /home/git/gitlab @@ -43,28 +46,31 @@ Check if GitLab and its dependencies are configured correctly: If all items are green, then congratulations upgrade is complete! -## 5. Upgrade GitLab Shell (if needed) - -If the `gitlab:check` task reports an outdated version of `gitlab-shell` you should upgrade it. +## 5. Upgrade GitLab Shell -Upgrade it by running the commands below after replacing 2.0.1 with the correct version number: +GitLab Shell might be outdated, running the commands below ensures you're using a compatible version: ``` cd /home/git/gitlab-shell sudo -u git -H git fetch -sudo -u git -H git checkout v2.0.1 +sudo -u git -H git checkout v`cat /home/git/gitlab/GITLAB_SHELL_VERSION` ``` ## One line upgrade command You've read through the entire guide and probably already did all the steps one by one. -Here is a one line command with step 1 to 4 for the next time you upgrade: +Here is a one line command with step 1 to 5 for the next time you upgrade: ```bash -cd /home/git/gitlab; sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production; \ +cd /home/git/gitlab; \ + sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production; \ sudo service gitlab stop; \ if [ -f bin/upgrade.rb ]; then sudo -u git -H ruby bin/upgrade.rb -y; else sudo -u git -H ruby script/upgrade.rb -y; fi; \ + cd /home/git/gitlab-shell; \ + sudo -u git -H git fetch; \ + sudo -u git -H git checkout v`cat /home/git/gitlab/GITLAB_SHELL_VERSION`; \ + cd /home/git/gitlab; \ sudo service gitlab start; \ sudo service nginx restart; sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production ``` diff --git a/doc/web_hooks/web_hooks.md b/doc/web_hooks/web_hooks.md index 31791da8074..851f50f5e9a 100644 --- a/doc/web_hooks/web_hooks.md +++ b/doc/web_hooks/web_hooks.md @@ -16,24 +16,29 @@ Triggered when you push to the repository except when pushing tags. ```json { + "object_kind": "push", "before": "95790bf891e76fee5e1747ab589903a6a1f80f22", "after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", "ref": "refs/heads/master", "user_id": 4, "user_name": "John Smith", + "user_email": "john@example.com", "project_id": 15, "repository": { "name": "Diaspora", - "url": "git@example.com:diaspora.git", + "url": "git@example.com:mike/diasporadiaspora.git", "description": "", - "homepage": "http://example.com/diaspora" + "homepage": "http://example.com/mike/diaspora", + "git_http_url":"http://example.com/mike/diaspora.git", + "git_ssh_url":"git@example.com:mike/diaspora.git", + "visibility_level":0 }, "commits": [ { "id": "b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327", "message": "Update Catalan translation to e38cb41.", "timestamp": "2011-12-12T14:27:31+02:00", - "url": "http://example.com/diaspora/commits/b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327", + "url": "http://example.com/mike/diaspora/commit/b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327", "author": { "name": "Jordi Mallach", "email": "jordi@softcatala.org" @@ -43,7 +48,7 @@ Triggered when you push to the repository except when pushing tags. "id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", "message": "fixed readme", "timestamp": "2012-01-03T23:36:29+02:00", - "url": "http://example.com/diaspora/commits/da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "url": "http://example.com/mike/diaspora/commit/da1560886d4f094c3e6c9ef40349f7d38b5d27d7", "author": { "name": "GitLab dev user", "email": "gitlabdev@dv6700.(none)" @@ -54,6 +59,35 @@ Triggered when you push to the repository except when pushing tags. } ``` +## Tag events + +Triggered when you create (or delete) tags to the repository. + +**Request body:** + +```json +{ + "object_kind": "tag_push", + "ref": "refs/tags/v1.0.0", + "before": "0000000000000000000000000000000000000000", + "after": "82b3d5ae55f7080f1e6022629cdb57bfae7cccc7", + "user_id": 1, + "user_name": "John Smith", + "project_id": 1, + "repository": { + "name": "jsmith", + "url": "ssh://git@example.com/jsmith/example.git", + "description": "", + "homepage": "http://example.com/jsmith/example", + "git_http_url":"http://example.com/jsmith/example.git", + "git_ssh_url":"git@example.com:jsmith/example.git", + "visibility_level":0 + }, + "commits": [], + "total_commits_count": 0 +} +``` + ## Issues events Triggered when a new issue is created or an existing issue was updated/closed/reopened. @@ -63,6 +97,11 @@ Triggered when a new issue is created or an existing issue was updated/closed/re ```json { "object_kind": "issue", + "user": { + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" + }, "object_attributes": { "id": 301, "title": "New API: create/update/delete file", @@ -92,6 +131,11 @@ Triggered when a new merge request is created or an existing merge request was u ```json { "object_kind": "merge_request", + "user": { + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" + }, "object_attributes": { "id": 99, "target_branch": "master", @@ -133,7 +177,9 @@ Triggered when a new merge request is created or an existing merge request was u "name": "GitLab dev user", "email": "gitlabdev@dv6700.(none)" } - } + }, + "url": "http://example.com/diaspora/merge_requests/1", + "action": "open" } } ``` diff --git a/doc/workflow/README.md b/doc/workflow/README.md index 323ee48f3bc..6e70235f5b8 100644 --- a/doc/workflow/README.md +++ b/doc/workflow/README.md @@ -1,6 +1,14 @@ -- [Workflow](workflow.md) +# Workflow + +- [Feature branch workflow](workflow.md) - [Project Features](project_features.md) - [Authorization for merge requests](authorization_for_merge_requests.md) - [Groups](groups.md) - [Labels](labels.md) - [GitLab Flow](gitlab_flow.md) +- [Notifications](notifications.md) +- [Migrating from SVN to GitLab](migrating_from_svn.md) +- [Project importing from GitHub to GitLab](import_projects_from_github.md) +- [Project importing from GitLab.com to your private GitLab instance](import_projects_from_gitlab_com.md) +- [Protected branches](protected_branches.md) +- [Web Editor](web_editor.md) diff --git a/doc/workflow/github_importer/importer.png b/doc/workflow/github_importer/importer.png Binary files differnew file mode 100644 index 00000000000..57636717571 --- /dev/null +++ b/doc/workflow/github_importer/importer.png diff --git a/doc/workflow/github_importer/new_project_page.png b/doc/workflow/github_importer/new_project_page.png Binary files differnew file mode 100644 index 00000000000..002f22d81d7 --- /dev/null +++ b/doc/workflow/github_importer/new_project_page.png diff --git a/doc/workflow/gitlab_flow.md b/doc/workflow/gitlab_flow.md index 70edea9c8d7..0e87dc74217 100644 --- a/doc/workflow/gitlab_flow.md +++ b/doc/workflow/gitlab_flow.md @@ -1,6 +1,6 @@  -# Introduction +## Introduction Version management with git makes branching and merging much easier than older versioning systems such as SVN. This allows a wide variety of branching strategies and workflows. @@ -26,12 +26,12 @@ After getting used to these three steps the branching model becomes the challeng Since many organizations new to git have no conventions how to work with it, it can quickly become a mess. The biggest problem they run into is that many long running branches that each contain part of the changes are around. People have a hard time figuring out which branch they should develop on or deploy to production. -Frequently the reaction to this problem is to adopt a standardized pattern such as [git flow](http://nvie.com/posts/a-successful-git-branching-model/) and [GitHub flow](https://guides.github.com/introduction/flow/index.html) +Frequently the reaction to this problem is to adopt a standardized pattern such as [git flow](http://nvie.com/posts/a-successful-git-branching-model/) and [GitHub flow](http://scottchacon.com/2011/08/31/github-flow.html) We think there is still room for improvement and will detail a set of practices we call GitLab flow. -# Git flow and its problems +## Git flow and its problems -[ +[ Git flow was one of the first proposals to use git branches and it has gotten a lot of attention. It advocates a master branch and a separate develop branch as well as supporting branches for features, releases and hotfixes. @@ -43,14 +43,14 @@ Since most tools automatically make the master branch the default one and displa The second problem of git flow is the complexity introduced by the hotfix and release branches. These branches can be a good idea for some organizations but are overkill for the vast majority of them. Nowadays most organizations practice continuous delivery which means that your default branch can be deployed. -This means that hotfixed and release branches can be prevented including all the ceremony they introduce. +This means that hotfix and release branches can be prevented including all the ceremony they introduce. An example of this ceremony is the merging back of release branches. Though specialized tools do exist to solve this, they require documentation and add complexity. Frequently developers make a mistake and for example changes are only merged into master and not into the develop branch. The root cause of these errors is that git flow is too complex for most of the use cases. And doing releases doesn't automatically mean also doing hotfixes. -# GitHub flow as a simpler alternative +## GitHub flow as a simpler alternative  @@ -62,13 +62,13 @@ Merging everything into the master branch and deploying often means you minimize But this flow still leaves a lot of questions unanswered regarding deployments, environments, releases and integrations with issues. With GitLab flow we offer additional guidance for these questions. -# Production branch with GitLab flow +## Production branch with GitLab flow  GitHub flow does assume you are able to deploy to production every time you merge a feature branch. This is possible for SaaS applications but are many cases where this is not possible. -One would be a situation where you are not in control of the exact release moment, for example an iOS application that needs to pass AppStore validation. +One would be a situation where you are not in control of the exact release moment, for example an iOS application that needs to pass App Store validation. Another example is when you have deployment windows (workdays from 10am to 4pm when the operations team is at full capacity) but you also merge code at other times. In these cases you can make a production branch that reflects the deployed code. You can deploy a new version by merging in master to the production branch. @@ -78,7 +78,7 @@ This time is pretty accurate if you automatically deploy your production branch. If you need a more exact time you can have your deployment script create a tag on each deployment. This flow prevents the overhead of releasing, tagging and merging that is common to git flow. -# Environment branches with GitLab flow +## Environment branches with GitLab flow  @@ -93,14 +93,14 @@ If master is good to go (it should be if you a practicing [continuous delivery]( If this is not possible because more manual testing is required you can send merge requests from the feature branch to the downstream branches. An 'extreme' version of environment branches are setting up an environment for each feature branch as done by [Teatro](http://teatro.io/). -# Release branches with GitLab flow +## Release branches with GitLab flow - + Only in case you need to release software to the outside world you need to work with release branches. In this case, each branch contains a minor version (2-3-stable, 2-4-stable, etc.). The stable branch uses master as a starting point and is created as late as possible. -By branching as late as possible you minimize the time you have to apply bugfixes to multiple branches. +By branching as late as possible you minimize the time you have to apply bug fixes to multiple branches. After a release branch is announced, only serious bug fixes are included in the release branch. If possible these bug fixes are first merged into master and then cherry-picked into the release branch. This way you can't forget to cherry-pick them into master and encounter the same bug on subsequent releases. @@ -109,7 +109,7 @@ Every time a bug-fix is included in a release branch the patch version is raised Some projects also have a stable branch that points to the same commit as the latest released branch. In this flow it is not common to have a production branch (or git flow master branch). -# Merge/pull requests with GitLab flow +## Merge/pull requests with GitLab flow  @@ -134,7 +134,7 @@ If the assigned person does not feel comfortable they can close the merge reques In GitLab it is common to protect the long-lived branches (e.g. the master branch) so that normal developers [can't modify these protected branches](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/permissions/permissions.md). So if you want to merge it into a protected branch you assign it to someone with master authorizations. -# Issues with GitLab flow +## Issues with GitLab flow  @@ -168,7 +168,7 @@ In this case it is no problem to reuse the same branch name since it was deleted At any time there is at most one branch for every issue. It is possible that one feature branch solves more than one issue. -# Linking and closing issues from merge requests +## Linking and closing issues from merge requests  @@ -177,11 +177,11 @@ In GitLab this creates a comment in the issue that the merge requests mentions t And the merge request shows the linked issues. These issues are closed once code is merged into the default branch. -If you only want to make the reference without closing the issue you can also just mention it: "Ducktyping is preferred. #12". +If you only want to make the reference without closing the issue you can also just mention it: "Duck typing is preferred. #12". If you have an issue that spans across multiple repositories, the best thing is to create an issue for each repository and link all issues to a parent issue. -# Squashing commits with rebase +## Squashing commits with rebase  @@ -189,7 +189,7 @@ With git you can use an interactive rebase (rebase -i) to squash multiple commit This functionality is useful if you made a couple of commits for small changes during development and want to replace them with a single commit or if you want to make the order more logical. However you should never rebase commits you have pushed to a remote server. Somebody can have referred to the commits or cherry-picked them. -When you rebase you change the identifier (SHA1) of the commit and this is confusing. +When you rebase you change the identifier (SHA-1) of the commit and this is confusing. If you do that the same change will be known under multiple identifiers and this can cause much confusion. If people already reviewed your code it will be hard for them to review only the improvements you made since then if you have rebased everything into one commit. @@ -207,7 +207,7 @@ If you revert a merge and you change your mind, revert the revert instead of mer Being able to revert a merge is a good reason always to create a merge commit when you merge manually with the `--no-ff` option. Git management software will always create a merge commit when you accept a merge request. -# Do not order commits with rebase +## Do not order commits with rebase  @@ -231,8 +231,8 @@ The last reason for creating merge commits is having long lived branches that yo Martin Fowler, in [his article about feature branches](http://martinfowler.com/bliki/FeatureBranch.html) talks about this Continuous Integration (CI). At GitLab we are guilty of confusing CI with branch testing. Quoting Martin Fowler: "I've heard people say they are doing CI because they are running builds, perhaps using a CI server, on every branch with every commit. That's continuous building, and a Good Thing, but there's no integration, so it's not CI.". -The solution to prevent many merge commits is to keep your feature branches shortlived, the vast majority should take less than one day of work. -If your feature branches commenly take more than a day of work, look into ways to create smaller units of work and/or use [feature toggles](http://martinfowler.com/bliki/FeatureToggle.html). +The solution to prevent many merge commits is to keep your feature branches short-lived, the vast majority should take less than one day of work. +If your feature branches commonly take more than a day of work, look into ways to create smaller units of work and/or use [feature toggles](http://martinfowler.com/bliki/FeatureToggle.html). As for the long running branches that take more than one day there are two strategies. In a CI strategy you can merge in master at the start of the day to prevent painful merges at a later time. In a synchronization point strategy you only merge in from well defined points in time, for example a tagged release. @@ -244,7 +244,7 @@ Developing software happen in small messy steps and it is OK to have your histor You can use tools to view the network graphs of commits and understand the messy history that created your code. If you rebase code the history is incorrect, and there is no way for tools to remedy this because they can't deal with changing commit identifiers. -# Voting on merge requests +## Voting on merge requests  @@ -252,7 +252,7 @@ It is common to voice approval or disapproval by using +1 or -1 emoticons. In GitLab the +1 and -1 are aggregated and shown at the top of the merge request. As a rule of thumb anything that doesn't have two times more +1's than -1's is suspect and should not be merged yet. -# Pushing and removing branches +## Pushing and removing branches  @@ -266,7 +266,7 @@ This ensures that the branch overview in the repository management software show This also ensures that when someone reopens the issue a new branch with the same name can be used without problem. When you reopen an issue you need to create a new merge request. -# Committing often and with the right message +## Committing often and with the right message  @@ -282,7 +282,7 @@ Some words that are bad commit messages because they don't contain munch informa The word fix or fixes is also a red flag, unless it comes after the commit sentence and references an issue number. To see more information about the formatting of commit messages please see this great [blog post by Tim Pope](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html). -# Testing before merging +## Testing before merging  @@ -299,7 +299,7 @@ If there are no merge conflicts and the feature branches are short lived the ris If there are merge conflicts you merge the master branch into the feature branch and the CI server will rerun the tests. If you have long lived feature branches that last for more than a few days you should make your issues smaller. -# Merging in other code +## Merging in other code  diff --git a/doc/workflow/gitlab_importer/importer.png b/doc/workflow/gitlab_importer/importer.png Binary files differnew file mode 100644 index 00000000000..d2a286d8cac --- /dev/null +++ b/doc/workflow/gitlab_importer/importer.png diff --git a/doc/workflow/gitlab_importer/new_project_page.png b/doc/workflow/gitlab_importer/new_project_page.png Binary files differnew file mode 100644 index 00000000000..5e239208e1e --- /dev/null +++ b/doc/workflow/gitlab_importer/new_project_page.png diff --git a/doc/workflow/import_projects_from_github.md b/doc/workflow/import_projects_from_github.md new file mode 100644 index 00000000000..8644b4ffc73 --- /dev/null +++ b/doc/workflow/import_projects_from_github.md @@ -0,0 +1,13 @@ +# Project importing from GitHub to GitLab + +You can import your existing GitHub projects to GitLab. But keep in mind that it is possible only if +GitHub support is enabled on your GitLab instance. You can read more about GitHub support [here](http://doc.gitlab.com/ce/integration/github.html) +To get to the importer page you need to go to "New project" page. + + + +Click on the "Import project from GitHub" link and you will be redirected to GitHub for permission to access your projects. After accepting, you'll be automatically redirected to the importer. + + + +To import a project, you can simple click "Add". The importer will import your repository and issues. Once the importer is done, a new GitLab project will be created with your imported data.
\ No newline at end of file diff --git a/doc/workflow/import_projects_from_gitlab_com.md b/doc/workflow/import_projects_from_gitlab_com.md new file mode 100644 index 00000000000..f4c4e955d46 --- /dev/null +++ b/doc/workflow/import_projects_from_gitlab_com.md @@ -0,0 +1,18 @@ +# Project importing from GitLab.com to your private GitLab instance + +You can import your existing GitLab.com projects to your GitLab instance. But keep in mind that it is possible only if +GitLab support is enabled on your GitLab instance. +You can read more about Gitlab support [here](http://doc.gitlab.com/ce/integration/gitlab.html) +To get to the importer page you need to go to "New project" page. + + + +Click on the "Import projects from Gitlab.com" link and you will be redirected to GitLab.com +for permission to access your projects. After accepting, you'll be automatically redirected to the importer. + + + + + +To import a project, you can simple click "Import". The importer will import your repository and issues. +Once the importer is done, a new GitLab project will be created with your imported data.
\ No newline at end of file diff --git a/doc/workflow/migrating_from_svn.md b/doc/workflow/migrating_from_svn.md new file mode 100644 index 00000000000..485db4834e9 --- /dev/null +++ b/doc/workflow/migrating_from_svn.md @@ -0,0 +1,17 @@ +# Migrating from SVN to GitLab + +SVN stands for Subversion and is a version control system (VCS). +Git is a distributed version control system. + +There are some major differences between the two, for more information consult your favorite search engine. + +Git has tools for migrating SVN repositories to git, namely `git svn`. You can read more about this at +[git documentation pages](http://git-scm.com/book/en/Git-and-Other-Systems-Git-and-Subversion). + +Apart from the [official git documentation](http://git-scm.com/book/en/Git-and-Other-Systems-Migrating-to-Git) there is also +user created step by step guide for migrating from SVN to GitLab. + +[Benjamin New](https://github.com/leftclickben) wrote [a guide that shows how to do a migration](https://gist.github.com/leftclickben/322b7a3042cbe97ed2af). Mirrors can be found [here](https://gitlab.com/snippets/2168) and [here](https://gist.github.com/maxlazio/f1b593b0d00aa966e9ca). + +## Contribute to this guide +We welcome all contributions that would expand this guide with instructions on how to migrate from SVN and other version control systems. diff --git a/doc/workflow/notifications.md b/doc/workflow/notifications.md new file mode 100644 index 00000000000..17215de677e --- /dev/null +++ b/doc/workflow/notifications.md @@ -0,0 +1,71 @@ +# GitLab Notifications + +GitLab has notifications system in place to notify a user of events important for the workflow. + +## Notification settings + +Under user profile page you can find the notification settings. + + + +Notification settings are divided into three groups: + +* Global Settings +* Group Settings +* Project Settings + +Each of these settings have levels of notification: + +* Disabled - turns off notifications +* Participating - receive notifications from related resources +* Watch - receive notifications from projects or groups user is a member of +* Global - notifications as set at the global settings + +#### Global Settings + +Global Settings are at the bottom of the hierarchy. +Any setting set here will be overridden by a setting at the group or a project level. + +Group or Project settings can use `global` notification setting which will then use +anything that is set at Global Settings. + +#### Group Settings + +Group Settings are taking precedence over Global Settings but are on a level below Project Settings. +This means that you can set a different level of notifications per group while still being able +to have a finer level setting per project. +Organization like this is suitable for users that belong to different groups but don't have the +same need for being notified for every group they are member of. + +#### Project Settings + +Project Settings are at the top level and any setting placed at this level will take precedence of any +other setting. +This is suitable for users that have different needs for notifications per project basis. + +## Notification events + +Below is the table of events users can be notified of: + +| Event | Sent to | Settings level | +|------------------------------|-------------------------------------------------------------------|------------------------------| +| New SSH key added | User | Security email, always sent. | +| New email added | User | Security email, always sent. | +| New user created | User | Sent on user creation, except for omniauth (LDAP)| +| New issue created | Issue assignee [1], project members [2] | [1] not disabled, [2] higher than participating | +| User added to project | User | Sent when user is added to project | +| Project access level changed | User | Sent when user project access level is changed | +| User added to group | User | Sent when user is added to group | +| Project moved | Project members [1] | [1] not disabled | +| Group access level changed | User | Sent when user group access level is changed | +| Close issue | Issue author [1], issue assignee [2], project members [3] | [1] [2] not disabled, [3] higher than participating | +| Reassign issue | New issue assignee [1], old issue assignee [2] | [1] [2] not disabled | +| Reopen issue | Project members [1] | [1] higher than participating | +| New merge request | MR assignee [1] | [1] not disabled | +| Reassign merge request | New MR assignee [1], old MR assignee [2] | [1] [2] not disabled | +| Close merge request | MR author [1], MR assignee [2], project members [3] | [1] [2] not disabled, [3] higher than participating | +| Reopen merge request | Project members [1] | [1] higher than participating | +| Merge merge request | MR author [1], MR assignee [2], project members [3] | [1] [2] not disabled, [3] higher than participating | +| New comment | Mentioned users [1], users participating [2], project members [3] | [1] [2] not disabled, [3] higher than participating | + + diff --git a/doc/workflow/notifications/settings.png b/doc/workflow/notifications/settings.png Binary files differnew file mode 100644 index 00000000000..e5b50ee2494 --- /dev/null +++ b/doc/workflow/notifications/settings.png diff --git a/doc/workflow/protected_branches.md b/doc/workflow/protected_branches.md new file mode 100644 index 00000000000..805f7f8d35c --- /dev/null +++ b/doc/workflow/protected_branches.md @@ -0,0 +1,33 @@ +# Protected branches + +Permission in GitLab are fundamentally defined around the idea of having read or write permission to the repository and branches. + +To prevent people from messing with history or pushing code without review, we've created protected branches. + +A protected branch does three simple things: + +* it prevents pushes from everybody except users with Master permission +* it prevents anyone from force pushing to the branch +* it prevents anyone from deleting the branch + +You can make any branch a protected branch. GitLab makes the master branch a protected branch by default. + +To protect a branch, user needs to have at least a Master permission level, see [permissions document](permissions/permissions.md). + + + +Navigate to project settings page and select `protected branches`. From the `Branch` dropdown menu select the branch you want to protect. + +Some workflows, like [GitLab workflow](gitlab_flow.md), require all users with write access to submit a Merge request in order to get the code into a protected branch. + +Since Masters and Owners can already push to protected branches, that means Developers cannot push to protected branch and need to submit a Merge request. + +However, there are workflows where that is not needed and only protecting from force pushes and branch removal is useful. + +For those workflows, you can allow everyone with write access to push to a protected branch by selecting `Developers can push` check box. + +On already protected branches you can also allow developers to push to the repository by selecting the `Developers can push` check box. + + + + diff --git a/doc/workflow/protected_branches/protected_branches1.png b/doc/workflow/protected_branches/protected_branches1.png Binary files differnew file mode 100644 index 00000000000..5c2a3de5f70 --- /dev/null +++ b/doc/workflow/protected_branches/protected_branches1.png diff --git a/doc/workflow/protected_branches/protected_branches2.png b/doc/workflow/protected_branches/protected_branches2.png Binary files differnew file mode 100644 index 00000000000..2dca3541365 --- /dev/null +++ b/doc/workflow/protected_branches/protected_branches2.png diff --git a/doc/workflow/web_editor.md b/doc/workflow/web_editor.md new file mode 100644 index 00000000000..7fc8f96b9ec --- /dev/null +++ b/doc/workflow/web_editor.md @@ -0,0 +1,26 @@ +# GitLab Web Editor + +In GitLab you can create new files and edit existing files using our web editor. +This is especially useful if you don't have access to a command line or you just want to do a quick fix. +You can easily access the web editor, depending on the context. +Let's start from newly created project. + +Click on `Add a file` +to create the first file and open it in the web editor. + + + +Fill in a file name, some content, a commit message, branch name and press the commit button. +The file will be saved to the repository. + + + +You can edit any text file in a repository by pressing the edit button, when +viewing the file. + + + +Editing a file is almost the same as creating a new file, +with as addition the ability to preview your changes in a separate tab. Also you can save your change to another branch by filling out field `branch` + + diff --git a/doc/workflow/web_editor/edit_file.png b/doc/workflow/web_editor/edit_file.png Binary files differnew file mode 100644 index 00000000000..f480c69ac3e --- /dev/null +++ b/doc/workflow/web_editor/edit_file.png diff --git a/doc/workflow/web_editor/empty_project.png b/doc/workflow/web_editor/empty_project.png Binary files differnew file mode 100644 index 00000000000..6a049f6beaf --- /dev/null +++ b/doc/workflow/web_editor/empty_project.png diff --git a/doc/workflow/web_editor/new_file.png b/doc/workflow/web_editor/new_file.png Binary files differnew file mode 100644 index 00000000000..55ebd9e0257 --- /dev/null +++ b/doc/workflow/web_editor/new_file.png diff --git a/doc/workflow/web_editor/show_file.png b/doc/workflow/web_editor/show_file.png Binary files differnew file mode 100644 index 00000000000..9cafcb55109 --- /dev/null +++ b/doc/workflow/web_editor/show_file.png diff --git a/doc/workflow/workflow.md b/doc/workflow/workflow.md index ab29cfb670b..f70e41df842 100644 --- a/doc/workflow/workflow.md +++ b/doc/workflow/workflow.md @@ -1,4 +1,4 @@ -# Workflow +# Feature branch workflow 1. Clone project: |
