Add latest changes from gitlab-org/gitlab@master

4 files changed, 96 insertions, 33 deletions

diff --git a/doc/development/product_analytics/usage_ping.md b/doc/development/product_analytics/usage_ping.md
index adb9d87cf6d..cb4d48ba457 100644
--- a/doc/development/product_analytics/usage_ping.md
+++ b/doc/development/product_analytics/usage_ping.md
@@ -132,8 +132,8 @@ general guidelines around how to collect those, due to the individual nature of
 There are several types of counters which are all found in `usage_data.rb`:
 
 - **Ordinary Batch Counters:** Simple count of a given ActiveRecord_Relation
-- **Distinct Batch Counters:** Distinct count of a given ActiveRecord_Relation on given column
-- **Sum Batch Counters:** Sum the values of a given ActiveRecord_Relation on given column
+- **Distinct Batch Counters:** Distinct count of a given ActiveRecord_Relation in a given column
+- **Sum Batch Counters:** Sum the values of a given ActiveRecord_Relation in a given column
 - **Alternative Counters:** Used for settings and configurations
 - **Redis Counters:** Used for in-memory counts.
 
@@ -153,7 +153,15 @@ For GitLab.com, there are extremely large tables with 15 second query timeouts,
 | `merge_request_diff_files`   | 1082                   |
 | `events`                     | 514                    |
 
-There are two batch counting methods provided, `Ordinary Batch Counters` and `Distinct Batch Counters`. Batch counting requires indexes on columns to calculate max, min, and range queries. In some cases, a specialized index may need to be added on the columns involved in a counter.
+We have several batch counting methods available: 
+
+- `Ordinary Batch Counters`
+- `Distinct Batch Counters`
+- `Sum Batch Counters`
+- `Estimated Batch Counters`
+
+Batch counting requires indexes on columns to calculate max, min, and range queries. In some cases,
+you may need to add a specialized index on the columns involved in a counter.
 
 ### Ordinary Batch Counters
 
@@ -248,6 +256,76 @@ sum(Issue.group(:state_id), :weight))
 # returns => {1=>3542, 2=>6820}
 ```
 
+### Estimated Batch Counters
+
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48233) in GitLab 13.7.
+
+Estimated batch counter functionality handles `ActiveRecord::StatementInvalid` errors
+when used through the provided `estimate_batch_distinct_count` method.
+Errors return a value of `-1`.
+
+WARNING:
+This functionality estimates a distinct count of a specific ActiveRecord_Relation in a given column,
+which uses the [HyperLogLog](http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf) algorithm.
+As the HyperLogLog algorithm is probabilistic, the **results always includes error**.
+The highest encountered error rate is 4.9%. 
+
+When correctly used, the `estimate_batch_distinct_count` method enables efficient counting over
+columns that contain non-unique values, which can not be assured by other counters. 
+
+Method: [`estimate_batch_distinct_count(relation, column = nil, batch_size: nil, start: nil, finish: nil)`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/utils/usage_data.rb#L63)
+
+The method includes the following arguments:
+
+- `relation`: The ActiveRecord_Relation to perform the count.
+- `column`: The column to perform the distinct count. The default is the primary key.
+- `batch_size`: The default is 10,000, from `Gitlab::Database::PostgresHll::BatchDistinctCounter::DEFAULT_BATCH_SIZE`.
+- `start`: The custom start of the batch count, to avoid complex minimum calculations.
+- `finish`: The custom end of the batch count in order to avoid complex maximum calculations.
+
+The method includes the following prerequisites:
+
+1. The supplied `relation` must include the primary key defined as the numeric column.
+   For example: `id bigint NOT NULL`.
+1. The `estimate_batch_distinct_count` can handle a joined relation. To utilize its ability to
+   count non-unique columns, the joined relation **must NOT** have a one-to-many relationship,
+   such as `has_many :boards`.
+1. Both `start` and `finish` arguments should always represent primary key relationship values,
+   even if the estimated count refers to another column, for example:
+
+   ```ruby 
+     estimate_batch_distinct_count(::Note, :author_id, start: ::Note.minimum(:id), finish: ::Note.maximum(:id))
+   ```
+
+Examples:
+
+1. Simple execution of estimated batch counter, with only relation provided, returned value will represent estimated 
+   number of unique values in `id` column (which is the primary key) of `Project` relation:
+
+   ```ruby 
+     estimate_batch_distinct_count(::Project)
+   ```
+   
+1. Execution of estimated batch counter, where provided relation has applied additional filter (`.where(time_period)`), number of unique values is going to be estimated in custom column (`:author_id`), and parameters: `start` and `finish` together apply boundaries that defines range of provided relation that is going to be analyzed 
+
+   ```ruby 
+     estimate_batch_distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::Note.minimum(:id), finish: ::Note.maximum(:id))
+   ```
+   
+1. Execution of estimated batch counter with joined relation (`joins(:cluster)`), for a custom column (`'clusters.user_id'`):
+
+   ```ruby 
+     estimate_batch_distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id')
+   ```
+
+When instrumenting metric with usage of estimated batch counter please add `_estimated` suffix to its name, for example:
+
+```ruby
+  "counts": {
+    "ci_builds_estimated": estimate_batch_distinct_count(Ci::Build),
+    ...
+```
+
 ### Redis Counters
 
 Handles `::Redis::CommandError` and `Gitlab::UsageDataCounters::BaseCounter::UnknownEvent`
@@ -309,6 +387,10 @@ Examples of implementation:
 
 #### Redis HLL Counters
 
+WARNING:
+HyperLogLog (HLL) is a probabilistic algorithm and its **results always includes some small error**. According to [Redis documentation](https://redis.io/commands/pfcount), data from
+used HLL implementation is "approximated with a standard error of 0.81%". 
+
 With `Gitlab::UsageDataCounters::HLLRedisCounter` we have available data structures used to count unique values.
 
 Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PFCOUNT](https://redis.io/commands/pfcount).
@@ -783,8 +865,6 @@ appear to be associated to any of the services running, since they all appear to
 ## Aggregated metrics
 
 > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45979) in GitLab 13.6.
-> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default.
-> - It's enabled on GitLab.com.
 
 WARNING:
 This feature is intended solely for internal GitLab use.
diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md
index 2fb1c9aefa1..7ce91ecb093 100644
--- a/doc/user/group/saml_sso/index.md
+++ b/doc/user/group/saml_sso/index.md
@@ -212,7 +212,7 @@ When a user tries to sign in with Group SSO, GitLab attempts to find or create a
 To link SAML to your existing GitLab.com account:
 
 1. Sign in to your GitLab.com account.
-1. Locate and visit the **GitLab single sign-on URL** for the group you're signing in to. A group Admin can find this on the group's **Settings > SAML SSO** page. If the sign-in URL is configured, users can connect to the GitLab app from the Identity Provider.
+1. Locate and visit the **GitLab single sign-on URL** for the group you're signing in to. A group owner can find this on the group's **Settings > SAML SSO** page. If the sign-in URL is configured, users can connect to the GitLab app from the Identity Provider.
 1. Click **Authorize**.
 1. Enter your credentials on the Identity Provider if prompted.
 1. You are then redirected back to GitLab.com and should now have access to the group. In the future, you can use SAML to sign in to GitLab.com.
diff --git a/doc/user/profile/notifications.md b/doc/user/profile/notifications.md
index 8974505cf02..ad18e5e78fd 100644
--- a/doc/user/profile/notifications.md
+++ b/doc/user/profile/notifications.md
@@ -146,13 +146,16 @@ Users are notified of the following events:
 | New email added              | User                | Security email, always sent. |
 | Email changed                | User                | Security email, always sent. |
 | Password changed             | User                | Security email, always sent when user changes their own password |
-| Password changed by administrator | User | Security email, always sent when an administrator changes the password of another user |
+| Password changed by administrator | User           | Security email, always sent when an administrator changes the password of another user |
 | Two-factor authentication disabled | User          | Security email, always sent. |
 | New user created             | User                | Sent on user creation, except for OmniAuth (LDAP)|
 | User added to project        | User                | Sent when user is added to project |
 | Project access level changed | User                | Sent when user project access level is changed |
 | User added to group          | User                | Sent when user is added to group |
 | Group access level changed   | User                | Sent when user group access level is changed |
+| Personal Access Tokens expiring soon | User          | Security email, always sent. |
+<!-- Do not delete or lint this instance of future tense -->
+| Personal Access Tokens have expired | User         | Security email, always sent. |
 | Project moved                | Project members (1) | (1) not disabled             |
 | New release                  | Project members     | Custom notification          |
 
diff --git a/doc/user/project/service_desk.md b/doc/user/project/service_desk.md
index bc9daf67626..76156690fe7 100644
--- a/doc/user/project/service_desk.md
+++ b/doc/user/project/service_desk.md
@@ -140,15 +140,12 @@ To edit the custom email display name:
 ### Using custom email address
 
 > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2201) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.0.
-> - It was [deployed behind a feature flag](../feature_flags.md), disabled by default.
-> - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/284656) on GitLab 13.7.
-> - It's enabled on GitLab.com.
-> - It's recommended for production use.
-> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#disable-custom-email-address). **(CORE ONLY)**
+> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284656) in GitLab 13.8.
 
-If the `service_desk_email` feature flag is enabled, then you can
-create Service Desk issues by sending emails to the Service Desk email address.
-The default address has the following format: `project_contact+%{key}@example.com`.
+If the `service_desk_email` is configured, then you can create Service Desk
+issues by sending emails to the Service Desk email address. The default
+address has the following format:
+`project_contact+%{key}@example.com`.
 
 The `%{key}` part is used to find the project where the issue should be created. The
 `%{key}` part combines the path to the project and configurable project name suffix:
@@ -163,7 +160,7 @@ always use separate mailboxes. This is important, because emails picked from
 `service_desk_email` mailbox are processed by a different worker and it would
 not recognize `incoming_email` emails.
 
-You can add the following snippets to your configuration:
+To configure a custom email address for Service Desk, add the following snippets to your configuration file:
 
 - Example for installations from source:
 
@@ -216,23 +213,6 @@ As a result, a new Service Desk issue is created from this email in the `mygroup
 The configuration options are the same as for configuring
 [incoming email](../../administration/incoming_email.md#set-it-up).
 
-#### Disable custom email address **(CORE ONLY)**
-
-Service Desk custom email is under development but ready for production use.
-It is deployed behind a feature flag that is **enabled by default**.
-
-To disable it:
-
-```ruby
-Feature.disable(:service_desk_custom_address)
-```
-
-To enable it:
-
-```ruby
-Feature.enable(:service_desk_custom_address)
-```
-
 ## Using Service Desk
 
 There are a few ways Service Desk can be used.