Switch UI tests from phantomjs to chrome headless (#14421)

* Make sure xss entries is always an array.

* Update expected test files and submodule.

* Regenerate broken omnifixture dump.

* poc conversion to chrome headless for UI tests, single test works locally

* Get single test to pass on travis & local w/ headless chrome.

* Remove old diff viewer generation, and get output to look the same as before when there is a failure.

* Add global timeout & get ViewDataTableTest to pass.

* Convert BarGraph_spec.js.

* Convert EmptySite_spec.

* Update EvolutionGraph spec for chrome headless.

* Convert GoalsTable test & find/replace for common changes.

* Convert MeasurableManager.

* Another find & replace.

* Get Menus/OptOutForm to pass w/ chrome headless

* Convert PeriodSelector, PieGraph & PivotByDimension UI tests.

* undo unrelated debugging change

* Convert QuickAccess UI tests

* Converts ActionsDataTable UI tests

* reset viewport after each testsuite

* Converts RowEvolution UI tests

* Converts Theme UI tests

* Converts SiteSelector UI tests

* Converts CustomVariables UI test

* Converts DBStats UI test

* Ignore empty responses when calling api/controller

* Converts Dashboard UI tests

* Converts Live UI tests

* Converts SimpleUI tests

* Converts Installation UI tests

* Converts ImageGraph UI tests

* Converts Login UI tests

* Converts Marketplace UI tests

* Converts Insights UI tests

* Converts ReportExporting UI tests

* Converts UIIntegration tests

* Updates ViewDataTable UI tests

* Converts CoreUpdater UI tests

* Converts UsersManager UI tests

* Converts Morpheus UI tests

* Converts MultiSites UI tests

* Fix testEnvironment.callApi handling of array parameters

* Converts Overlay UI tests

* Converts PrivacyManager UI tests

* Converts ScheduledReports UI tests

* Converts SegmentEditor UI tests

* compare pixel difference in images

* Converts PeriodSelector UI tests

* allow a pixel difference

* Converts SingleMetricView UI tests

* Converts SitesManager UI tests

* Converts Transitions UI tests

* Converts MeasurableManager UI tests

* Move Mouse out of screen after each test suite

* Converts PieGraph UI tests

* Ensure cursor is not shown in focused input elements

* Converts VisitorMap UI tests

* Converts CustomLogo UI tests

* Converts BarGraph UI tests

* Converts EvolutionGraph UI tests

* Converts IntranetMeasurable UI tests

* Converts TrackingFailures UI tests

* Converts CampaignBuilder UI tests

* Converts TagManagerTeaser UI tests

* fix request url rewrite for changed port

* Converts TwoFactorAuth UI tests

* do not print full responses in logs, but size instead

* improve additional style injection

* use submodule branches

* fix request interception

* update dependencies

* [TEMP] disable dangerous link checks

* [TEMP] limit tests

* Use puppeteer@next and make sure diffs get uploaded properly.

* Use correct fs function.

* Allow timeout to be specified on command line for UI tests and debug console logs.

* timeout option tweak

* Setup diff dir once before running tests not before each suite.

* fix last commit

* Update screenshots and fix some specs.

* Convert some tests that were not converted during merge.

* updating screenshots

* Updating screenshots and fixing some tests.

* more test fixes

* couple more test fixes

* More test fixes and plugin updates.

* wait for jquery

* Fix some screenshots and tests.

* more test fixes

* debug travis test failure

* remove .only

* more test fixes and updated expected files

* another test fixing iteration

* typo

* another quick test change

* more tests updates

* Test fixes and & more debugging.

* more debugging and test fixes

* more fixes and debugging.

* remove .only

* Another round of fixed tests.

* more debugging & fixes

* more test changes

* more debugging and fixes

* Fixing more tests + some bug fixes.

* Couple more fixes.

* Couple more fixes.

* Fixing tracking failures test

* more fixes

* Last couple fixes hopefully.

* couple more test fixes

* more fixes, bug fix in usersmanager, some test updates

* Some more test fixes/changes.

* more and more fixes

* hoping for a green build

* Do not compare against existing files in dashboard_spec.

* more fixes, re-enabling travis

* really re-enable travis

* Install puppeteer on travis-ci.

* more fixes

* try to fix random failues

* remove return

* Convert some login tests I forgot and update UsersManager expected file

* Fix test for primary key update

* Update AddPrimaryKey.php

* More test fixes + implement synchronous comparison threshold (so no resemblejs)

* More wait fixes + reset token detection fix.

* couple more fixes

* Fix login/overlay screenshot issue & magick command running issue.

* identify is the executable on travis (hopefully) + fix some random failures

* Another run.

* Last fix.

* two more tweaks

* typo

6 files changed, 52 insertions, 44 deletions

diff --git a/plugins/UserCountryMap/Controller.php b/plugins/UserCountryMap/Controller.php
index 5bf9a49e37..7358a40487 100644
--- a/plugins/UserCountryMap/Controller.php
+++ b/plugins/UserCountryMap/Controller.php
@@ -15,6 +15,7 @@ use Piwik\Config;
 use Piwik\Container\StaticContainer;
 use Piwik\Piwik;
 use Piwik\Plugins\Goals\API as APIGoals;
+use Piwik\Plugins\VisitsSummary\API as VisitsSummaryAPI;
 use Piwik\Site;
 use Piwik\Translation\Translator;
 use Piwik\View;
@@ -66,7 +67,7 @@ class Controller extends \Piwik\Plugin\Controller
 
         // request visits summary
         $request = new Request(
-            'method=VisitsSummary.get&format=PHP'
+            'method=VisitsSummary.get&format=php'
             . '&idSite=' . $this->idSite
             . '&period=' . $period
             . '&date=' . $date
diff --git a/plugins/UserCountryMap/tests/UI/VisitorMap_spec.js b/plugins/UserCountryMap/tests/UI/VisitorMap_spec.js
index c8685a3e7f..64b11de2a0 100644
--- a/plugins/UserCountryMap/tests/UI/VisitorMap_spec.js
+++ b/plugins/UserCountryMap/tests/UI/VisitorMap_spec.js
@@ -15,46 +15,53 @@ describe("VisitorMap", function () {
         urlWithCities = "?module=Widgetize&action=iframe&moduleToWidgetize=UserCountryMap&idSite=3&period=week&date=yesterday&"
             + "actionToWidgetize=visitorMap&viewDataTable=table&filter_limit=5&isFooterExpandedInDashboard=1";
 
-    it("should display the bounce rate metric correctly", function (done) {
-        this.retries(3);
-
-        expect.screenshot('bounce_rate').to.be.similar(0.002).to.capture(function (page) {
-            page.load(url);
-            page.evaluate(function () {
-                $('.userCountryMapSelectMetrics').val('bounce_rate').trigger('change');
-            });
-            page.mouseMove('.UserCountryMap_map.kartograph');
-        }, done);
+    it("should display the bounce rate metric correctly", async function() {
+        await page.goto(url);
+        await page.evaluate(function () {
+            $('.userCountryMapSelectMetrics').val('bounce_rate').trigger('change');
+        });
+        await page.mouse.move(900, 140);
+        await page.waitFor(100); // wait for tooltip
+
+        expect(await page.screenshot({ fullPage: true })).to.matchImage('bounce_rate');
     });
 
-    it("should display the average time on site metric correctly", function (done) {
-        expect.screenshot('avg_time_on_site').to.be.similar(0.002).to.capture(function (page) {
-            page.evaluate(function () {
-                $('.userCountryMapSelectMetrics').val('avg_time_on_site').trigger('change');
-            });
-            page.mouseMove('.UserCountryMap_map.kartograph');
-        }, done);
+    it("should display the average time on site metric correctly", async function() {
+        await page.mouse.move(0, 0);
+        await page.evaluate(function () {
+            $('.userCountryMapSelectMetrics').val('avg_time_on_site').trigger('change');
+        });
+        await page.mouse.move(900, 140);
+        await page.waitFor(100); // wait for tooltip
+
+        expect(await page.screenshot({ fullPage: true })).to.matchImage('avg_time_on_site');
     });
 
-    it("should display the regions layer correctly", function (done) {
-        expect.screenshot('regions').to.be.similar(0.002).to.capture(function (page) {
-            page.load(urlWithCities);
-            page.evaluate(function () {
-                // zoom into USA
-                var path = window.visitorMap.map.getLayer('countries').getPaths({iso: "USA"})[0].svgPath[0];
-                $(path).click();
-            });
-            page.evaluate(function () {
-                // go to regions view
-                var path = window.visitorMap.map.getLayer('countries').getPaths({iso: "USA"})[0].svgPath[0];
-                $(path).click();
-            });
-        }, done);
+    it("should display the regions layer correctly", async function() {
+        await page.goto(urlWithCities);
+        await page.waitForNetworkIdle();
+        await page.waitFor(1000);
+        await page.webpage.evaluate(function () {
+            // zoom into USA
+            var path = window.visitorMap.map.getLayer('countries').getPaths({iso: "USA"})[0].svgPath[0];
+            $(path).click();
+        });
+        await page.waitFor(1000);
+        await page.webpage.evaluate(function () {
+            // go to regions view
+            var path = window.visitorMap.map.getLayer('countries').getPaths({iso: "USA"})[0].svgPath[0];
+            $(path).click();
+        });
+        await page.waitFor(1000);
+
+        expect(await page.screenshot({ fullPage: true })).to.matchImage('regions');
     });
 
-    it("should display the cities layer correctly", function (done) {
-        expect.screenshot('cities').to.be.similar(0.002).to.capture(function (page) {
-            page.click('.UserCountryMap-btn-city');
-        }, done);
+    it("should display the cities layer correctly", async function() {
+        await page.click('.UserCountryMap-btn-city');
+        await page.waitForNetworkIdle();
+        await page.waitFor(1000); // wait for map
+
+        expect(await page.screenshot({ fullPage: true })).to.matchImage('cities');
     });
 });
diff --git a/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_avg_time_on_site.png b/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_avg_time_on_site.png
index ceb7af09ad..7427536122 100644
--- a/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_avg_time_on_site.png
+++ b/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_avg_time_on_site.png
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b700da83b5b12979d23b2f2d068d6ecf27595ddcffcf8082363c0d4ebec3de17
-size 189886
+oid sha256:46359162b1e148d84803672849a9f321e6da2bad79fe5d7ed4adfb32f255c0ee
+size 182721
diff --git a/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_bounce_rate.png b/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_bounce_rate.png
index c4026194f2..b501e348a1 100644
--- a/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_bounce_rate.png
+++ b/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_bounce_rate.png
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:43f1e81a0a440b37126fbc028a971e742b103721f59a4ce6b5f362fbb904b0d2
-size 189092
+oid sha256:c448f1190057040db9f83063a7130b419797228c030ae31590484674b83e7e87
+size 182099
diff --git a/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_cities.png b/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_cities.png
index 30a9ce43e4..5037ce92d4 100644
--- a/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_cities.png
+++ b/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_cities.png
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9f3f0e40d23b460a7efca8b67d312418ccb95e17db83913fdec72037291aabf0
-size 117136
+oid sha256:58558ea7698b691511024179fe61c560934bbe006c4fac0ad69e5e0378e8313c
+size 114807
diff --git a/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_regions.png b/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_regions.png
index 0ccb5cf81e..9079033b43 100644
--- a/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_regions.png
+++ b/plugins/UserCountryMap/tests/UI/expected-screenshots/VisitorMap_regions.png
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:23c6affafe3db866251746f771e2c2fe48d09150fceb4d8f1c60bcdfb4596337
-size 116052
+oid sha256:bda0d6a54b07a55a9e34514c44d101a76ed89d4493f121bf3ba70a5d964c8997
+size 114263