GDPR + Add a couple of new platform features especially around validating settings (#12558)

* add a couple of new platform features especially around validating settings

* more validators and other tweaks

* few tweaks

* allow plugins to add header script in JS tests

* some work on gdpr

* remove email feature

* menu update

* add date tooltip

* make sure privacy has its own icon

* added some text

* add opt out info

* better titles

* add some information about webserver logs

* add log import notice

* give angular some time to render

* show border around iframe

* make sure to place a notification in modal dialog if needed

* when opening a modal, make sure the top part is visible

* disable newsletter sign up by default

* command line tool to anonymize and remove previously tracked raw data

* implement some basic consent features into piwik.js

* minor tweak

* schedule entries to anonymize data in past

* let users anonymize historical data

* better overview

* better naming

* explain consent tracking

* add legitimate interest

* new menu structure for privacy

* use a proper db table for scheduled anonymizations

* add some tracker tests

* some tests and fixes

* couple of fixes and tests

* starting to add some UI tests

* more tweaks

* add missing file

* fix jslint

* have the field full width

* more tweaks

* create directory if needed

* various fixes

* fix some tests

* make sure keys will be always returned in same order

* more fixes

* fix some tests

* apply feedback from review

* fix some tests

* this might fix mysqli tests

* fix some tests

* trying to fix mysqli tests

* fix system test

* fix comment

* remove some duplicated code

* update method name

* trigger event when logs need to be deleted

* add possibility to anonymize userId and orderId

* trigger event in tracking code page

* expire consent in hours

* post another event

* update wording for privacy GDPR

* fix various ui tests

* fix some tests

* trying to fix ui tests

* trigger another test run

* fix system test

* fix some ui tests

* show a table of contents on tracking code page, fix title, add mobile apps and API

* fix ui test

* 3.5.0-b2 update

* Re-generate piwik.js

* fix job status and selector style

* make sure to include only sites with admin access

* some tweaks

* fix some ui tests

* fix some ui tests

* fix test

5 files changed, 278 insertions, 35 deletions

diff --git a/plugins/PrivacyManager/templates/askingForConsent.twig b/plugins/PrivacyManager/templates/askingForConsent.twig
new file mode 100644
index 0000000000..ffd4975e5d
--- /dev/null
+++ b/plugins/PrivacyManager/templates/askingForConsent.twig
@@ -0,0 +1,105 @@
+{% extends 'admin.twig' %}
+
+{% set title %}{{ 'PrivacyManager_GDPR'|translate }}{% endset %}
+
+{% block content %}
+    <div piwik-content-intro>
+        <h2>Asking for consent</h2>
+        <p>Consent means offering individuals choice and control about how their personal data is processed.</p>
+    </div>
+
+    <div piwik-content-block content-title="When do I need to ask for user consent?" class="privacyAskingForConsent">
+        <p>
+            In the context of the GDPR privacy regulations, when you are processing personal data, in some cases you will need to ask for your users' consent. To identify whether you need to ask for consent, you need to determine whether your lawful basis for processing personal data is "Consent" or "Legitimate interest", or whether you can avoid collecting personal data altogether. We recommend to learn more about the lawful basis under the GDPR for Matomo.
+            <br /><br />
+            Note that when you require to have your users' consents and if you cannot prove that you got the "Consent" of your users in the past, then we recommend to <a href="{{ linkTo({module: 'PrivacyManager', action: 'privacySettings'}) }}">anonymise your previously tracked users' raw data</a>.
+        </p>
+    </div>
+
+    <div piwik-content-block content-title="How do I ask users for consent?" class="privacyAskingForConsent">
+        <p>
+            Follow the steps below to ask your user for their consents before their data is processed within Matomo.
+        </p>
+
+        <h3>Step 1: require consent</h3>
+        <p>
+            To require consent, insert the following line at top of your existing Matomo Tracking code on all your pages:
+        </p>
+        <pre piwik-select-on-focus><code>// require user consent before processing data
+_paq.push(['requireConsent']);
+_paq.push([‘trackPageview']);
+[...]</code></pre>
+        <p>
+            Once this function ‘requireConsent' is executed then no tracking request will be sent to Matomo. Note that some <a href="https://matomo.org/faq/general/faq_146/" target="_blank" rel="noreferrer noopener">tracking cookies</a> will be set during this page view but will be deleted as soon as the user navigates to a different page or closes the browser.
+        </p>
+        <h3>Step 2: asking for consent through your privacy notice</h3>
+        <p>Now you can ask the user for consent for example by displaying a clear privacy notice on your pages. Learn more about privacy notices and asking for user consent. Note that Matomo does not yet offer the feature to display a privacy notice, but may implement such a feature in the future to easily let you display the notice and gather user consent.
+        </p>
+        <h3>Step 3: user gives consent</h3>
+        <p>
+            Once a user gives consent, you can either A) let Matomo remember the consent, or B) use your own consent tool to remember the consent. We present the two solutions below:
+            <br /><br />
+            <strong>A) if you want to let Matomo remember the consent</strong>
+            <br />
+            Once a user gives their consent, you can let Matomo remember that the user has given consent by simply calling the following method once the user has given her or his consent:
+        </p>
+        <pre piwik-select-on-focus><code>_paq.push(['rememberConsentGiven'])</code></pre>
+        <p>
+            Matomo will then remember on subsequent requests that the user has given her or his consent by setting a cookie named "consent". As long as this cookie exists, Matomo will know that consent has been given and will automatically process the data. This means that you only need to call <code piwik-select-on-focus>_paq.push(['rememberConsentGiven'])</code> once, and it is not needed to call <code piwik-select-on-focus>_paq.push(['setConsentGiven'])</code> on any page view.
+            <br /><br />
+            Notes:
+        </p>
+            <ol>
+            <li>By default, the cookie and consent will be remembered forever. It is possible to define an optional expiry period for your user consent by calling:
+
+                <br /><code piwik-select-on-focus>_paq.push(['rememberConsentGiven', optionallyExpireConsentInHours])</code>.</li>
+            <li>When you're tracking multiple sub-domains into the same website in Matomo, you want to ensure that when you ask for Consent, the user gives consent for all the sub-domains on which you are collecting data. If the user only gives consent for a particular domain or sub-domain(s), you may need to restrict or widen the scope of the consent cookie domain and path by using 'setCookieDomain' and ‘setCookiePath' (learn more in the JavaScript tracking guide). </li>
+            <li>It is required that you do not disable cookies.</li>
+            </ol>
+        <p>
+            <br />
+            <br />
+            <strong>B) if you use your own consent tool to remember the consent</strong>
+            <br />
+            In some cases, you record the information that the user has given consent to be tracked directly in your own system or CMS (for example when you use your own a cookie to remember user consent). Once you have the consent by the user to process their data, you need to call the 'setConsentGiven' method:
+        </p>
+        <pre piwik-select-on-focus><code>// require user consent before processing data
+_paq.push(['requireConsent']);
+_paq.push([‘trackPageview']);
+[...]
+
+// user has given consent to process their data
+_paq.push(['setConsentGiven']);</code></pre>
+        <p>
+            This lets the JavaScript tracker know that the user has given consent and ensures the tracking is working as expected. This function needs to be called anytime after <code piwik-select-on-focus>_paq.push(['requireConsent'])</code>.
+            <br />
+            <br />
+            Notes:
+        </p>
+        <ol>
+            <li>when you call <code piwik-select-on-focus>_paq.push(['setConsentGiven'])</code>, Matomo will not remember on subsequent requests that this user has given consent: it is important that you call setConsentGiven on every page.</li>
+            <li>when the user has given consent, you could also avoid calling <code piwik-select-on-focus>_paq.push(['requireConsent'])</code> in the first place. </li>
+        </ol>
+
+        <h3>Step 4: user removes consent</h3>
+        <p>
+            In order to remove his consent the user needs to perform a specific action, for example: clicking on a button "I do not want to be tracked anymore".
+            <br /><br />
+            <strong>A) if you want to let Matomo remember the consent</strong>
+            <br />
+            When the user has expressed she or he no longer gives consent, you need to call the following method once:
+        </p>
+        <pre piwik-select-on-focus><code> _paq.push(['forgetConsentGiven']);</code></pre>
+        <p>This makes sure the cookie that remembered the given consent is deleted.
+
+            <br />
+            <br />
+            <strong>B) if you use your own consent tool to remember the consent</strong><br />
+            When the user has expressed she or he no longer gives consent, you shall not call the following method anymore:
+            <br />
+        </p>
+        <pre piwik-select-on-focus><code>// do not call this once user has removed their consent
+_paq.push(['setConsentGiven']);</code></pre>
+
+    </div>
+{% endblock %}
diff --git a/plugins/PrivacyManager/templates/gdprOverview.twig b/plugins/PrivacyManager/templates/gdprOverview.twig
new file mode 100644
index 0000000000..899e661799
--- /dev/null
+++ b/plugins/PrivacyManager/templates/gdprOverview.twig
@@ -0,0 +1,55 @@
+{% extends 'admin.twig' %}
+
+{% set title %}{{ 'PrivacyManager_GDPR'|translate }}{% endset %}
+
+{% block content %}
+<div class="gdprOverview">
+    <div piwik-content-intro>
+        <h2>GDPR overview</h2>
+        <p>
+            The General Data Protection Regulation (GDPR) is a regulation which strengthen and unify data protection for all individuals within the European Union (EU).
+<br /><br />
+            If you take steps to ensure no personal data is collected in Matomo then you may not be concerned by the GDPR (when you track no IP address, no user IDs, no geolocation data, etc.).
+
+        </p>
+    </div>
+
+    <div piwik-content-block content-title="GDPR checklists">
+        <p>If you are processing personal data of European citizens through Matomo, even if your company is located outside Europe, you need to fulfill GDPR obligations and this guide will help you.
+<br /><br />
+            Find below our tools that let you exercise your users’ rights easily, and the list of actions to take in order to make your use of Matomo compliant with the GDPR and safeguard your data. Visit our <a rel="noreferrer noopener" target="_blank" href="https://matomo.org/docs/gdpr">GDPR User guide</a> to learn even more.</p>
+    </div>
+    <div piwik-content-block content-title="Individuals' rights">
+        <p>Exercise the rights of your users with our GDPR-friendly procedures:
+        </p>
+        <ol>
+            <li>The right to be informed: inform your users with a clear privacy notice.</li>
+            <li>The right of access: <a target="_blank" rel="noreferrer noopener" href="{{ linkTo({module: 'PrivacyManager', action: 'gdprTools'}) }}">search for a data subject</a> and export all of their data.</li>
+            <li>The right to erasure: <a target="_blank" rel="noreferrer noopener" href="{{ linkTo({module: 'PrivacyManager', action: 'gdprTools'}) }}">search for a data subject</a> and delete some or all of their data.</li>
+            <li>The right to rectification: you can <a target="_blank" rel="noreferrer noopener" href="{{ linkTo({module: 'PrivacyManager', action: 'gdprTools'}) }}">search for a data subject</a> and delete some or all of their data.</li>
+            <li>The right to data portability: <a target="_blank" rel="noreferrer noopener" href="{{ linkTo({module: 'PrivacyManager', action: 'gdprTools'}) }}">search for a data subject</a> and export all of their data.</li>
+            <li>The right to object: <a target="_blank" rel="noreferrer noopener" href="{{ linkTo({module: 'PrivacyManager', action: 'usersOptOut'}) }}">let your users easily opt-out</a> in your privacy policy page.</li>
+            <li>If you offer online services to children and relies on consent to collect information about them, then you may need a parent or guardian’s consent in order to process the children’s personal data lawfully.</li>
+        </ol>
+    </div>
+
+    <div piwik-content-block content-title="Awareness & documentation">
+        <p>Inform your users clearly and transparently, and make your colleagues aware of the data being collected and how it is used:</p>
+        <ol>
+            <li>Inform your visitors through a clear privacy notice whenever you’re collecting personal data.</li>
+            <li>Inform your users in your privacy policy about what data you collect and how the data is used.</li>
+            <li>Make your team aware that you are using Matomo Analytics and <a target="_blank" rel="noreferrer noopener" href="https://matomo.org/faq/general/faq_18254/">what data is being collected by your analytics platform</a>.</li>
+            <li>Document your use of Matomo within your <a href="https://matomo.org/blog/2018/04/gdpr-how-to-fill-in-the-information-asset-register-when-using-matomo/" target="_blank" rel="noreferrer noopener">information asset register</a>.</li>
+        </ol>
+    </div>
+    <div piwik-content-block content-title="Security procedures">
+        <p>Inform your users clearly and transparently, and make your colleagues aware of the data being collected and how it is used:</p>
+        <ol>
+            <li>Apply our <a href="https://matomo.org/docs/security/" target="_blank" rel="noreferrer noopener">security recommendations</a> in order to keep your Matomo data safe.</li>
+            <li>Check that you have a written contract with the company providing you the Matomo server or hosting which ensures <a href="https://ico.org.uk/for-organisations/guide-to-the-general-data-protection-regulation-gdpr/international-transfers/" target="_blank" rel="noreferrer noopener">appropriate safeguards are provided</a>.</li>
+            <li>Include Matomo in your <a href="https://ico.org.uk/for-organisations/guide-to-the-general-data-protection-regulation-gdpr/personal-data-breaches/" target="_blank" rel="noreferrer noopener">data breach procedure</a>.</li>
+            <li>Include Matomo in your <a href="https://www.cnil.fr/en/guidelines-dpia" target="_blank" rel="noreferrer noopener">data privacy impact assessment (DPIA)</a>, if applicable.</li>
+        </ol>
+    </div>
+</div>
+{% endblock %}
diff --git a/plugins/PrivacyManager/templates/gdprTools.twig b/plugins/PrivacyManager/templates/gdprTools.twig
new file mode 100644
index 0000000000..228c1cb240
--- /dev/null
+++ b/plugins/PrivacyManager/templates/gdprTools.twig
@@ -0,0 +1,7 @@
+{% extends 'admin.twig' %}
+
+{% set title %}{{ 'PrivacyManager_GDPR'|translate }}{% endset %}
+
+{% block content %}
+    <div matomo-manage-gdpr></div>
+{% endblock %}
diff --git a/plugins/PrivacyManager/templates/privacySettings.twig b/plugins/PrivacyManager/templates/privacySettings.twig
index 11fa148e5e..3aa8a85401 100644
--- a/plugins/PrivacyManager/templates/privacySettings.twig
+++ b/plugins/PrivacyManager/templates/privacySettings.twig
@@ -1,21 +1,20 @@
 {% extends 'admin.twig' %}
 
-{% set title %}{{ 'PrivacyManager_TeaserHeadline'|translate }}{% endset %}
+{% set title %}{{ 'PrivacyManager_AnonymizeData'|translate }}{% endset %}
 
 {% block content %}
 {% import 'macros.twig' as piwik %}
-{% if isSuperUser %}
 
 <div piwik-content-intro>
     <h2 piwik-enriched-headline help-url="https://matomo.org/docs/privacy/">{{ title }}</h2>
 
-    <p>{{ 'PrivacyManager_Teaser'|translate('<a href="#anonymizeIPAnchor">',"</a>",'<a href="#deleteLogsAnchor">',"</a>",'<a href="#optOutAnchor">',"</a>")|raw }}
+    <p>{{ 'PrivacyManager_TeaserHeader'|translate('<a href="#anonymizeIPAnchor">',"</a>",'<a href="#deleteLogsAnchor">',"</a>",'<a href="#anonymizeHistoricalData">',"</a>")|raw }}
         {{'PrivacyManager_SeeAlsoOurOfficialGuidePrivacy'|translate('<a href="https://matomo.org/privacy/" rel="noreferrer"  target="_blank">','</a>')|raw }}</p>
 </div>
 
 <div piwik-content-block
      id="anonymizeIPAnchor"
-     content-title="{{ 'PrivacyManager_UseAnonymizeIp'|translate|e('html_attr') }}">
+     content-title="{{ 'PrivacyManager_UseAnonymizeTrackingData'|translate|e('html_attr') }}">
     <div piwik-form ng-controller="AnonymizeIpController as anonymizeIp">
 
         <div piwik-field uicontrol="checkbox" name="anonymizeIpSettings"
@@ -43,6 +42,20 @@
             </div>
         </div>
 
+        <div piwik-field uicontrol="checkbox" name="anonymizeUserId"
+             ng-model="anonymizeIp.anonymizeUserId"
+             title="{{ 'PrivacyManager_UseAnonymizeUserId'|translate|e('html_attr') }}"
+             value="{{ anonymizeIP.anonymizeUserId }}"
+             inline-help="{{ 'PrivacyManager_AnonymizeUserIdNote'|translate|e('html_attr') }}">
+        </div>
+
+        <div piwik-field uicontrol="checkbox" name="anonymizeOrderId"
+             ng-model="anonymizeIp.anonymizeOrderId"
+             title="{{ 'PrivacyManager_UseAnonymizeOrderId'|translate|e('html_attr') }}"
+             value="{{ anonymizeIP.anonymizeOrderId }}"
+             inline-help="{{ 'PrivacyManager_AnonymizeOrderIdNote'|translate|e('html_attr') }}">
+        </div>
+
         <div piwik-save-button onconfirm="anonymizeIp.save()" saving="anonymizeIp.isLoading"></div>
     </div>
 </div>
@@ -276,41 +289,62 @@
         </div>
     {% endif %}
 </div>
-<div piwik-content-block
-    id="DNT"
-    content-title="{{ 'PrivacyManager_DoNotTrack_SupportDNTPreference'|translate|e('html_attr') }}">
-    <p>
-        {% if dntSupport %}
-            <strong>{{ 'PrivacyManager_DoNotTrack_Enabled'|translate }}</strong>
-            <br/>
-            {{ 'PrivacyManager_DoNotTrack_EnabledMoreInfo'|translate }}
-        {% else %}
-            {{ 'PrivacyManager_DoNotTrack_Disabled'|translate }} {{ 'PrivacyManager_DoNotTrack_DisabledMoreInfo'|translate }}
-        {% endif %}
-    </p>
 
-    <div piwik-form ng-controller="DoNotTrackPreferenceController as doNotTrack">
+    <a name="anonymizeHistoricalData" id="anonymizeHistoricalData"></a>
 
-        {# {{ {'module':'PrivacyManager','nonce':nonce,'action':action} | urlRewriteWithParameters }}#DNT #}
-        <div piwik-field uicontrol="radio" name="doNotTrack"
-             ng-model="doNotTrack.enabled"
-             options="{{ doNotTrackOptions|json_encode }}"
-             value="{% if dntSupport %}1{% else %}0{% endif %}"
-             inline-help="{{ 'PrivacyManager_DoNotTrack_Description'|translate|e('html_attr') }}">
-        </div>
+    <div piwik-content-block content-title="Anonymize previously tracked raw data" class="logDataAnonymizer">
+        <p>If you have tracked personal data such as the full visitor IP, you may
+            want to anonymize this data now in case you do not have consent for this data or no longer a legitimate
+            interest.
+        </p>
 
-        <div piwik-save-button onconfirm="doNotTrack.save()" saving="doNotTrack.isLoading"></div>
+        {% if isSuperUser %}
+            <div matomo-anonymize-log-data></div>
+        {% else %}
+            <p>Only a user with Super User access can anonymize previously tracked raw data.</p>
+        {% endif %}
 
+        <br />
+        <h3>Previous log data anonymizations</h3>
+        <table piwik-content-table>
+            <thead>
+            <tr>
+                <th>Requester</th>
+                <th>Affected ID Sites</th>
+                <th>Affected date</th>
+                <th>Anonymize</th>
+                <th>Visit Columns</th>
+                <th>Link Visit Action Columns</th>
+                <th>Status</th>
+            </tr></thead>
+            <tbody>
+            {% for entry in anonymizations %}
+                {% set isStarted = entry.job_start_date or not entry.job_start_date is empty %}
+                {% set isFinished = entry.job_finish_date or not entry.job_finish_date is empty %}
+                <tr>
+                    <td>{{ entry.requester }}</td>
+                    <td>{{ entry.sites|join(', ') }}</td>
+                    <td>{{ entry.date_start }} - {{ entry.date_end }}</td>
+                    <td>{% if entry.anonymize_ip %}<span>IP address<br /></span>{% endif %}
+                        {% if entry.anonymize_location %}<span>Location<br /></span>{% endif %}
+                        {% if entry.anonymize_userid %}<span>User ID</span>{% endif %}
+                        {% if not entry.anonymize_ip and not entry.anonymize_location and not entry.anonymize_userid %}-{% endif %}
+                    </td>
+                    <td>{{ entry.unset_visit_columns|join(', ') }}</td>
+                    <td>{{ entry.unset_link_visit_action_columns|join(', ') }}</td>
+                    <td>
+                        {% if not isStarted %}
+                            <span class="icon-info" style="cursor: help;" title="Scheduled date: {{ entry.scheduled_date|default('') }}."></span> Scheduled
+                        {% elseif isStarted and not isFinished %}
+                            <span class="icon-info" style="cursor: help;" title="Scheduled date: {{ entry.scheduled_date|default('') }}. Job Start Date: {{ entry.job_start_date }}. Current Output: {{ entry.output }}"></span> In progress
+                        {% else %}
+                            <span class="icon-info" style="cursor: help;" title="Scheduled date: {{ entry.scheduled_date|default('') }}. Job Start Date: {{ entry.job_start_date }}. Job Finish Date: {{ entry.job_finish_date }}. Output: {{ entry.output }}"></span> Done
+                        {% endif %}
+                    </td>
+                </tr>
+            {% endfor %}
+            </tbody>
+        </table>
     </div>
 
-
-{% endif %}
-</div>
-<div piwik-content-block
-     id="optOutAnchor"
-     content-title="{{ 'CoreAdminHome_OptOutForYourVisitors'|translate|e('html_attr') }}">
-    <div piwik-opt-out-customizer piwikurl="{{ piwikUrl }}" language="{{ language }}">
-    
-    </div>
-</div>
 {% endblock %}
diff --git a/plugins/PrivacyManager/templates/usersOptOut.twig b/plugins/PrivacyManager/templates/usersOptOut.twig
new file mode 100644
index 0000000000..f0615d87c1
--- /dev/null
+++ b/plugins/PrivacyManager/templates/usersOptOut.twig
@@ -0,0 +1,42 @@
+{% extends 'admin.twig' %}
+
+{% set title %}{{ 'PrivacyManager_GDPR'|translate }}{% endset %}
+
+{% block content %}
+    <div piwik-content-block content-title="Let users opt-out of tracking">
+        <div piwik-opt-out-customizer piwikurl="{{ piwikUrl }}" language="{{ language }}">
+
+        </div>
+    </div>
+
+    {% if isSuperUser %}
+        <div piwik-content-block
+             id="DNT"
+             content-title="{{ 'PrivacyManager_DoNotTrack_SupportDNTPreference'|translate|e('html_attr') }}">
+            <p>
+                {% if dntSupport %}
+                    <strong>{{ 'PrivacyManager_DoNotTrack_Enabled'|translate }}</strong>
+                    <br/>
+                    {{ 'PrivacyManager_DoNotTrack_EnabledMoreInfo'|translate }}
+                {% else %}
+                    {{ 'PrivacyManager_DoNotTrack_Disabled'|translate }} {{ 'PrivacyManager_DoNotTrack_DisabledMoreInfo'|translate }}
+                {% endif %}
+            </p>
+
+            <div piwik-form ng-controller="DoNotTrackPreferenceController as doNotTrack">
+
+                {# {{ {'module':'PrivacyManager','nonce':nonce,'action':action} | urlRewriteWithParameters }}#DNT #}
+                <div piwik-field uicontrol="radio" name="doNotTrack"
+                     ng-model="doNotTrack.enabled"
+                     options="{{ doNotTrackOptions|json_encode }}"
+                     value="{% if dntSupport %}1{% else %}0{% endif %}"
+                     inline-help="{{ 'PrivacyManager_DoNotTrack_Description'|translate|e('html_attr') }}">
+                </div>
+
+                <div piwik-save-button onconfirm="doNotTrack.save()" saving="doNotTrack.isLoading"></div>
+
+            </div>
+
+        </div>
+    {% endif %}
+{% endblock %}