Merge branch 'dm-remove-private-token' into 'master'

Remove Private Tokens

Closes #38595 and #38447

See merge request gitlab-org/gitlab-ce!14838

9 files changed, 79 insertions, 142 deletions

diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md
index 6f1356ddf8f..be538ea250a 100644
--- a/doc/administration/troubleshooting/debug.md
+++ b/doc/administration/troubleshooting/debug.md
@@ -141,7 +141,7 @@ separate Rails process to debug the issue:
 
 1. Log in to your GitLab account.
 1. Copy the URL that is causing problems (e.g. https://gitlab.com/ABC).
-1. Obtain the private token for your user (Profile Settings -> Account).
+1. Create a Personal Access Token for your user (Profile Settings -> Access Tokens).
 1. Bring up the GitLab Rails console. For omnibus users, run:
 
     ```
diff --git a/doc/api/README.md b/doc/api/README.md
index 89ffe9d7868..f226716c3b5 100644
--- a/doc/api/README.md
+++ b/doc/api/README.md
@@ -50,7 +50,6 @@ following locations:
 - [Repository Files](repository_files.md)
 - [Runners](runners.md)
 - [Services](services.md)
-- [Session](session.md)
 - [Settings](settings.md)
 - [Sidekiq metrics](sidekiq_metrics.md)
 - [System Hooks](system_hooks.md)
@@ -86,27 +85,10 @@ API requests should be prefixed with `api` and the API version. The API version
 is defined in [`lib/api.rb`][lib-api-url]. For example, the root of the v4 API
 is at `/api/v4`.
 
-For endpoints that require [authentication](#authentication), you need to pass
-a `private_token` parameter via query string or header. If passed as a header,
-the header name must be `PRIVATE-TOKEN` (uppercase and with a dash instead of
-an underscore).
-
-Example of a valid API request:
-
-```
-GET /projects?private_token=9koXpg98eAheJpvBs5tK
-```
-
-Example of a valid API request using cURL and authentication via header:
+Example of a valid API request using cURL:
 
 ```shell
-curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects"
-```
-
-Example of a valid API request using cURL and authentication via a query string:
-
-```shell
-curl "https://gitlab.example.com/api/v4/projects?private_token=9koXpg98eAheJpvBs5tK"
+curl "https://gitlab.example.com/api/v4/projects"
 ```
 
 The API uses JSON to serialize data. You don't need to specify `.json` at the
@@ -114,15 +96,20 @@ end of an API URL.
 
 ## Authentication
 
-Most API requests require authentication via a session cookie or token. For
+Most API requests require authentication, or will only return public data when
+authentication is not provided. For
 those cases where it is not required, this will be mentioned in the documentation
 for each individual endpoint. For example, the [`/projects/:id` endpoint](projects.md).
 
-There are three types of access tokens available:
+There are three ways to authenticate with the GitLab API:
 
 1. [OAuth2 tokens](#oauth2-tokens)
-1. [Private tokens](#private-tokens)
 1. [Personal access tokens](#personal-access-tokens)
+1. [Session cookie](#session-cookie)
+
+For admins who want to authenticate with the API as a specific user, or who want to build applications or scripts that do so, two options are available:
+1. [Impersonation tokens](#impersonation-tokens)
+2. [Sudo](#sudo)
 
 If authentication information is invalid or omitted, an error message will be
 returned with status code `401`:
@@ -133,74 +120,84 @@ returned with status code `401`:
 }
 ```
 
-### Session cookie
+### OAuth2 tokens
 
-When signing in to GitLab as an ordinary user, a `_gitlab_session` cookie is
-set. The API will use this cookie for authentication if it is present, but using
-the API to generate a new session cookie is currently not supported.
+You can use an [OAuth2 token](oauth2.md) to authenticate with the API by passing it in either the
+`access_token` parameter or the `Authorization` header.
 
-### OAuth2 tokens
+Example of using the OAuth2 token in a parameter:
 
-You can use an OAuth 2 token to authenticate with the API by passing it either in the
-`access_token` parameter or in the `Authorization` header.
+```shell
+curl https://gitlab.example.com/api/v4/projects?access_token=OAUTH-TOKEN
+```
 
-Example of using the OAuth2 token in the header:
+Example of using the OAuth2 token in a header:
 
 ```shell
 curl --header "Authorization: Bearer OAUTH-TOKEN" https://gitlab.example.com/api/v4/projects
 ```
 
-Read more about [GitLab as an OAuth2 client](oauth2.md).
+Read more about [GitLab as an OAuth2 provider](oauth2.md).
 
-### Private tokens
+### Personal access tokens
 
-Private tokens provide full access to the GitLab API. Anyone with access to
-them can interact with GitLab as if they were you. You can find or reset your
-private token in your account page (`/profile/account`).
+You can use a [personal access token][pat] to authenticate with the API by passing it in either the
+`private_token` parameter or the `Private-Token` header.
 
-For examples of usage, [read the basic usage section](#basic-usage).
+Example of using the personal access token in a parameter:
 
-### Personal access tokens
+```shell
+curl https://gitlab.example.com/api/v4/projects?private_token=9koXpg98eAheJpvBs5tK
+```
+
+Example of using the personal access token in a header:
 
-Instead of using your private token which grants full access to your account,
-personal access tokens could be a better fit because of their granular
-permissions.
+```shell
+curl --header "Private-Token: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects
+```
 
-Once you have your token, pass it to the API using either the `private_token`
-parameter or the `PRIVATE-TOKEN` header. For examples of usage,
-[read the basic usage section](#basic-usage).
+Read more about [personal access tokens][pat].
+
+### Session cookie
+
+When signing in to the main GitLab application, a `_gitlab_session` cookie is
+set. The API will use this cookie for authentication if it is present, but using
+the API to generate a new session cookie is currently not supported.
 
-[Read more about personal access tokens.][pat]
+The primary user of this authentication method is the web frontend of GitLab itself,
+which can use the API as the authenticated user to get a list of their projects,
+for example, without needing to explicitly pass an access token.
 
 ### Impersonation tokens
 
 > [Introduced][ce-9099] in GitLab 9.0. Needs admin permissions.
 
 Impersonation tokens are a type of [personal access token][pat]
-that can only be created by an admin for a specific user.
+that can only be created by an admin for a specific user. They are a great fit
+if you want to build applications or scripts that authenticate with the API as a specific user.
 
-They are a better alternative to using the user's password/private token
-or using the [Sudo](#sudo) feature which also requires the admin's password
-or private token, since the password/token can change over time. Impersonation
-tokens are a great fit if you want to build applications or tools which
-authenticate with the API as a specific user.
+They are an alternative to directly using the user's password or one of their
+personal access tokens, and to using the [Sudo](#sudo) feature, since the user's (or admin's, in the case of Sudo)
+password/token may not be known or may change over time.
 
 For more information, refer to the
 [users API](users.md#retrieve-user-impersonation-tokens) docs.
 
-For examples of usage, [read the basic usage section](#basic-usage).
+Impersonation tokens are used exactly like regular personal access tokens, and can be passed in either the
+`private_token` parameter or the `Private-Token` header.
 
 ### Sudo
 
 > Needs admin permissions.
 
 All API requests support performing an API call as if you were another user,
-provided your private token is from an administrator account. You need to pass
-the `sudo` parameter either via query string or a header with an ID/username of
+provided you are authenticated as an administrator with an OAuth or Personal Access Token that has the `sudo` scope.
+
+You need to pass the `sudo` parameter either via query string or a header with an ID/username of
 the user you want to perform the operation as. If passed as a header, the
-header name must be `SUDO` (uppercase).
+header name must be `Sudo`.
 
-If a non administrative `private_token` is provided, then an error message will
+If a non administrative access token is provided, an error message will
 be returned with status code `403`:
 
 ```json
@@ -209,12 +206,23 @@ be returned with status code `403`:
 }
 ```
 
+If an access token without the `sudo` scope is provided, an error message will
+be returned with status code `403`:
+
+```json
+{
+  "error": "insufficient_scope",
+  "error_description": "The request requires higher privileges than provided by the access token.",
+  "scope": "sudo"
+}
+```
+
 If the sudo user ID or username cannot be found, an error message will be
 returned with status code `404`:
 
 ```json
 {
-  "message": "404 Not Found: No user id or username for: <id/username>"
+  "message": "404 User with ID or username '123' Not Found"
 }
 ```
 
@@ -228,7 +236,7 @@ GET /projects?private_token=9koXpg98eAheJpvBs5tK&sudo=username
 ```
 
 ```shell
-curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --header "SUDO: username" "https://gitlab.example.com/api/v4/projects"
+curl --header "Private-Token: 9koXpg98eAheJpvBs5tK" --header "Sudo: username" "https://gitlab.example.com/api/v4/projects"
 ```
 
 Example of a valid API call and a request using cURL with sudo request,
@@ -239,7 +247,7 @@ GET /projects?private_token=9koXpg98eAheJpvBs5tK&sudo=23
 ```
 
 ```shell
-curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --header "SUDO: 23" "https://gitlab.example.com/api/v4/projects"
+curl --header "Private-Token: 9koXpg98eAheJpvBs5tK" --header "Sudo: 23" "https://gitlab.example.com/api/v4/projects"
 ```
 
 ## Status codes
diff --git a/doc/api/session.md b/doc/api/session.md
deleted file mode 100644
index b97e26f34a2..00000000000
--- a/doc/api/session.md
+++ /dev/null
@@ -1,55 +0,0 @@
-# Session API
-
->**Deprecation notice:**
-Starting in GitLab 8.11, this feature has been **disabled** for users with
-[two-factor authentication][2fa] turned on. These users can access the API
-using [personal access tokens] instead.
-
-You can login with both GitLab and LDAP credentials in order to obtain the
-private token.
-
-```
-POST /session
-```
-
-| Attribute  | Type    | Required | Description |
-| ---------- | ------- | -------- | -------- |
-| `login`    | string  | yes      | The username of the user|
-| `email`    | string  | yes if login is not provided | The email of the user |
-| `password` | string  | yes     | The password of the user |
-
-```bash
-curl --request POST "https://gitlab.example.com/api/v4/session?login=john_smith&password=strongpassw0rd"
-```
-
-Example response:
-
-```json
-{
-  "name": "John Smith",
-  "username": "john_smith",
-  "id": 32,
-  "state": "active",
-  "avatar_url": null,
-  "created_at": "2015-01-29T21:07:19.440Z",
-  "is_admin": true,
-  "bio": null,
-  "skype": "",
-  "linkedin": "",
-  "twitter": "",
-  "website_url": "",
-  "email": "john@example.com",
-  "theme_id": 1,
-  "color_scheme_id": 1,
-  "projects_limit": 10,
-  "current_sign_in_at": "2015-07-07T07:10:58.392Z",
-  "identities": [],
-  "can_create_group": true,
-  "can_create_project": true,
-  "two_factor_enabled": false,
-  "private_token": "9koXpg98eAheJpvBs5tK"
-}
-```
-
-[2fa]: ../user/profile/account/two_factor_authentication.md
-[personal access tokens]: ../user/profile/personal_access_tokens.md
diff --git a/doc/api/users.md b/doc/api/users.md
index 1643c584244..aa711090af1 100644
--- a/doc/api/users.md
+++ b/doc/api/users.md
@@ -410,8 +410,7 @@ GET /user
   "can_create_group": true,
   "can_create_project": true,
   "two_factor_enabled": true,
-  "external": false,
-  "private_token": "dd34asd13as"
+  "external": false
 }
 ```
 
diff --git a/doc/development/doc_styleguide.md b/doc/development/doc_styleguide.md
index 798f40eef3d..0e4ffbd7910 100644
--- a/doc/development/doc_styleguide.md
+++ b/doc/development/doc_styleguide.md
@@ -459,11 +459,11 @@ Rendered example:
 ### cURL commands
 
 - Use `https://gitlab.example.com/api/v4/` as an endpoint.
-- Wherever needed use this private token: `9koXpg98eAheJpvBs5tK`.
+- Wherever needed use this personal access token: `9koXpg98eAheJpvBs5tK`.
 - Always put the request first. `GET` is the default so you don't have to
   include it.
 - Use double quotes to the URL when it includes additional parameters.
-- Prefer to use examples using the private token and don't pass data of
+- Prefer to use examples using the personal access token and don't pass data of
   username and password.
 
 | Methods | Description |
diff --git a/doc/raketasks/user_management.md b/doc/raketasks/user_management.md
index 3ae46019daf..5554a0c8b78 100644
--- a/doc/raketasks/user_management.md
+++ b/doc/raketasks/user_management.md
@@ -149,18 +149,3 @@ cp config/secrets.yml.bak config/secrets.yml
 sudo /etc/init.d/gitlab start
 
 ```
-
-## Clear authentication tokens for all users. Important! Data loss!
-
-Clear authentication tokens for all users in the GitLab database. This
-task is useful if your users' authentication tokens might have been exposed in
-any way. All the existing tokens will become invalid, and new tokens are
-automatically generated upon sign-in or user modification.
-
-```
-# omnibus-gitlab
-sudo gitlab-rake gitlab:users:clear_all_authentication_tokens
-
-# installation from source
-bundle exec rake gitlab:users:clear_all_authentication_tokens RAILS_ENV=production
-```
diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md
index 5561784ed0b..042cde3f01e 100644
--- a/doc/topics/autodevops/index.md
+++ b/doc/topics/autodevops/index.md
@@ -517,7 +517,7 @@ Feature.get(:auto_devops_banner_disabled).enable
 Or through the HTTP API with an admin access token:
 
 ```sh
-curl --data "value=true" --header "PRIVATE-TOKEN: private_token" https://gitlab.example.com/api/v4/features/auto_devops_banner_disabled
+curl --data "value=true" --header "PRIVATE-TOKEN: personal_access_token" https://gitlab.example.com/api/v4/features/auto_devops_banner_disabled
 ```
 
 [ce-37115]: https://gitlab.com/gitlab-org/gitlab-ce/issues/37115
diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md
index 5ebb88bf324..5fcc0501dc1 100644
--- a/doc/user/profile/index.md
+++ b/doc/user/profile/index.md
@@ -52,7 +52,7 @@ You can edit your account settings by navigating from the up-right corner menu b
 From there, you can:
 
 - Update your personal information
-- Manage [private tokens](../../api/README.md#private-tokens), email tokens, [2FA](account/two_factor_authentication.md)
+- Manage [2FA](account/two_factor_authentication.md)
 - Change your username and [delete your account](account/delete_account.md)
 - Manage applications that can
 [use GitLab as an OAuth provider](../../integration/oauth_provider.md#introduction-to-oauth)
diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md
index f28c034e74c..9b4fdd65e2f 100644
--- a/doc/user/profile/personal_access_tokens.md
+++ b/doc/user/profile/personal_access_tokens.md
@@ -2,17 +2,15 @@
 
 > [Introduced][ce-3749] in GitLab 8.8.
 
-Personal access tokens are useful if you need access to the [GitLab API][api].
-Instead of using your private token which grants full access to your account,
-personal access tokens could be a better fit because of their
-[granular permissions](#limiting-scopes-of-a-personal-access-token).
+Personal access tokens are the preferred way for third party applications and scripts to
+authenticate with the [GitLab API][api], if using [OAuth2](../../api/oauth2.md) is not practical.
 
 You can also use them to authenticate against Git over HTTP. They are the only
 accepted method of authentication when you have
 [Two-Factor Authentication (2FA)][2fa] enabled.
 
 Once you have your token, [pass it to the API][usage] using either the
-`private_token` parameter or the `PRIVATE-TOKEN` header.
+`private_token` parameter or the `Private-Token` header.
 
 The expiration of personal access tokens happens on the date you define,
 at midnight UTC.
@@ -49,12 +47,14 @@ the following table.
 |`read_user` | Allows access to the read-only endpoints under `/users`. Essentially, any of the `GET` requests in the [Users API][users] are allowed ([introduced][ce-5951] in GitLab 8.15). |
 | `api` | Grants complete access to the API (read/write) ([introduced][ce-5951] in GitLab 8.15). Required for accessing Git repositories over HTTP when 2FA is enabled. |
 | `read_registry` | Allows to read [container registry] images if a project is private and authorization is required ([introduced][ce-11845] in GitLab 9.3). |
+| `sudo` | Allows performing API actions as any user in the system (if the authenticated user is an admin) ([introduced][ce-14838] in GitLab 10.2). |
 
 [2fa]: ../account/two_factor_authentication.md
 [api]: ../../api/README.md
 [ce-3749]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3749
 [ce-5951]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5951
 [ce-11845]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11845
+[ce-14838]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14838
 [container registry]: ../project/container_registry.md
 [users]: ../../api/users.md
-[usage]: ../../api/README.md#basic-usage
+[usage]: ../../api/README.md#personal-access-tokens