Switch UI tests from phantomjs to chrome headless (#14421)

* Make sure xss entries is always an array.

* Update expected test files and submodule.

* Regenerate broken omnifixture dump.

* poc conversion to chrome headless for UI tests, single test works locally

* Get single test to pass on travis & local w/ headless chrome.

* Remove old diff viewer generation, and get output to look the same as before when there is a failure.

* Add global timeout & get ViewDataTableTest to pass.

* Convert BarGraph_spec.js.

* Convert EmptySite_spec.

* Update EvolutionGraph spec for chrome headless.

* Convert GoalsTable test & find/replace for common changes.

* Convert MeasurableManager.

* Another find & replace.

* Get Menus/OptOutForm to pass w/ chrome headless

* Convert PeriodSelector, PieGraph & PivotByDimension UI tests.

* undo unrelated debugging change

* Convert QuickAccess UI tests

* Converts ActionsDataTable UI tests

* reset viewport after each testsuite

* Converts RowEvolution UI tests

* Converts Theme UI tests

* Converts SiteSelector UI tests

* Converts CustomVariables UI test

* Converts DBStats UI test

* Ignore empty responses when calling api/controller

* Converts Dashboard UI tests

* Converts Live UI tests

* Converts SimpleUI tests

* Converts Installation UI tests

* Converts ImageGraph UI tests

* Converts Login UI tests

* Converts Marketplace UI tests

* Converts Insights UI tests

* Converts ReportExporting UI tests

* Converts UIIntegration tests

* Updates ViewDataTable UI tests

* Converts CoreUpdater UI tests

* Converts UsersManager UI tests

* Converts Morpheus UI tests

* Converts MultiSites UI tests

* Fix testEnvironment.callApi handling of array parameters

* Converts Overlay UI tests

* Converts PrivacyManager UI tests

* Converts ScheduledReports UI tests

* Converts SegmentEditor UI tests

* compare pixel difference in images

* Converts PeriodSelector UI tests

* allow a pixel difference

* Converts SingleMetricView UI tests

* Converts SitesManager UI tests

* Converts Transitions UI tests

* Converts MeasurableManager UI tests

* Move Mouse out of screen after each test suite

* Converts PieGraph UI tests

* Ensure cursor is not shown in focused input elements

* Converts VisitorMap UI tests

* Converts CustomLogo UI tests

* Converts BarGraph UI tests

* Converts EvolutionGraph UI tests

* Converts IntranetMeasurable UI tests

* Converts TrackingFailures UI tests

* Converts CampaignBuilder UI tests

* Converts TagManagerTeaser UI tests

* fix request url rewrite for changed port

* Converts TwoFactorAuth UI tests

* do not print full responses in logs, but size instead

* improve additional style injection

* use submodule branches

* fix request interception

* update dependencies

* [TEMP] disable dangerous link checks

* [TEMP] limit tests

* Use puppeteer@next and make sure diffs get uploaded properly.

* Use correct fs function.

* Allow timeout to be specified on command line for UI tests and debug console logs.

* timeout option tweak

* Setup diff dir once before running tests not before each suite.

* fix last commit

* Update screenshots and fix some specs.

* Convert some tests that were not converted during merge.

* updating screenshots

* Updating screenshots and fixing some tests.

* more test fixes

* couple more test fixes

* More test fixes and plugin updates.

* wait for jquery

* Fix some screenshots and tests.

* more test fixes

* debug travis test failure

* remove .only

* more test fixes and updated expected files

* another test fixing iteration

* typo

* another quick test change

* more tests updates

* Test fixes and & more debugging.

* more debugging and test fixes

* more fixes and debugging.

* remove .only

* Another round of fixed tests.

* more debugging & fixes

* more test changes

* more debugging and fixes

* Fixing more tests + some bug fixes.

* Couple more fixes.

* Couple more fixes.

* Fixing tracking failures test

* more fixes

* Last couple fixes hopefully.

* couple more test fixes

* more fixes, bug fix in usersmanager, some test updates

* Some more test fixes/changes.

* more and more fixes

* hoping for a green build

* Do not compare against existing files in dashboard_spec.

* more fixes, re-enabling travis

* really re-enable travis

* Install puppeteer on travis-ci.

* more fixes

* try to fix random failues

* remove return

* Convert some login tests I forgot and update UsersManager expected file

* Fix test for primary key update

* Update AddPrimaryKey.php

* More test fixes + implement synchronous comparison threshold (so no resemblejs)

* More wait fixes + reset token detection fix.

* couple more fixes

* Fix login/overlay screenshot issue & magick command running issue.

* identify is the executable on travis (hopefully) + fix some random failures

* Another run.

* Last fix.

* two more tweaks

* typo

8 files changed, 59 insertions, 59 deletions

diff --git a/plugins/CoreUpdater/tests/UI/CoreUpdaterCode_spec.js b/plugins/CoreUpdater/tests/UI/CoreUpdaterCode_spec.js
index 70b55dbbe9..682f92f9d9 100644
--- a/plugins/CoreUpdater/tests/UI/CoreUpdaterCode_spec.js
+++ b/plugins/CoreUpdater/tests/UI/CoreUpdaterCode_spec.js
@@ -14,27 +14,26 @@ describe("CoreUpdaterCode", function () {
 
     var url = "?module=CoreUpdater&action=newVersionAvailable";
 
-    it("should show a new version is available", function (done) {
-        expect.screenshot("newVersion").to.be.capture(function (page) {
-            page.load(url);
-        }, done);
+    it("should show a new version is available", async function() {
+        await page.goto(url);
+        expect(await page.screenshot({ fullPage: true })).to.matchImage('newVersion');
     });
 
-    it("should offer to retry using https when updating over https fails", function (done) {
-        expect.screenshot("httpsUpdateFail").to.be.capture(function (page) {
-            page.click('#updateAutomatically');
-        }, done);
+    it("should offer to retry using https when updating over https fails", async function() {
+        await page.click('#updateAutomatically');
+        await page.waitForNetworkIdle();
+        expect(await page.screenshot({ fullPage: true })).to.matchImage('httpsUpdateFail');
     });
 
-    it("should offer to retry over http when updating over https fails", function (done) {
-        expect.screenshot("httpsUpdateFail").to.be.capture(function (page) {
-            page.click('#updateUsingHttps');
-        }, done);
+    it("should offer to retry over http when updating over https fails", async function() {
+        await page.click('#updateUsingHttps');
+        await page.waitForNetworkIdle();
+        expect(await page.screenshot({ fullPage: true })).to.matchImage('httpsUpdateFail');
     });
 
-    it("should show the update steps when updating over http succeeds", function (done) {
-        expect.screenshot("httpUpdateSuccess").to.be.capture(function (page) {
-            page.click('#updateUsingHttp');
-        }, done);
+    it("should show the update steps when updating over http succeeds", async function() {
+        await page.click('#updateUsingHttp');
+        await page.waitForNetworkIdle();
+        expect(await page.screenshot({ fullPage: true })).to.matchImage('httpUpdateSuccess');
     });
 });
diff --git a/plugins/CoreUpdater/tests/UI/CoreUpdaterDb_spec.js b/plugins/CoreUpdater/tests/UI/CoreUpdaterDb_spec.js
index 3fb5595c78..114b95d31a 100644
--- a/plugins/CoreUpdater/tests/UI/CoreUpdaterDb_spec.js
+++ b/plugins/CoreUpdater/tests/UI/CoreUpdaterDb_spec.js
@@ -25,49 +25,50 @@ describe("CoreUpdaterDb", function () {
     });
 
     function apiUpgradeTest(format) {
-        it("should start the updater when an old version of Piwik is detected in the DB with format " + format, function (done) {
-            expect.file('CoreUpdater.API.ErrorMessage' + format + '.txt').to.be.pageContents(function (page) {
-                page.load('');
-                page.downloadUrl('?module=API&method=API.getPiwikVersion&format=' + format);
-            }, done);
+        it("should start the updater when an old version of Piwik is detected in the DB with format " + format, async function() {
+            const url = '?module=API&method=API.getPiwikVersion&format=' + format;
+            var pageContents = await page.downloadUrl(url);
+
+            expect.file('CoreUpdater.API.ErrorMessage' + format + '.txt').to.equal(pageContents);
         });
     }
 
     var formats = ['CSV', 'TSV', 'XML', 'JSON', 'PHP'];
     formats.forEach(apiUpgradeTest);
 
-    it("should start the updater when an old version of Piwik is detected in the DB", function (done) {
-        expect.screenshot("main").to.be.capture(function (page) {
-            page.load("");
-            page.evaluate(function () {
-                $('p').each(function () {
-                    var replace = $(this).html().replace(/(?!1\.0)\d+\.\d+(\.\d+)?([\-a-z]*\d+)?/g, '');
-                    $(this).html(replace);
-                });
+    it("should start the updater when an old version of Piwik is detected in the DB", async function() {
+        await page.goto("");
+        await page.evaluate(function () {
+            $('p').each(function () {
+                var replace = $(this).html().replace(/(?!1\.0)\d+\.\d+(\.\d+)?([\-a-z]*\d+)?/g, '');
+                $(this).html(replace);
             });
-        }, done);
+        });
+
+        expect(await page.screenshot({ fullPage: true })).to.matchImage('main');
     });
 
-    it("should show instance id in updating screen", function (done) {
-        expect.screenshot("main_instance").to.be.capture(function (page) {
-            testEnvironment.configOverride.General = {
-                instance_id: 'custom.instance'
-            };
-            testEnvironment.save();
+    it("should show instance id in updating screen", async function() {
+        testEnvironment.configOverride.General = {
+            instance_id: 'custom.instance'
+        };
+        testEnvironment.save();
 
-            page.load("");
-            page.evaluate(function () {
-                $('p').each(function () {
-                    var replace = $(this).html().replace(/(?!1\.0)\d+\.\d+(\.\d+)?([\-a-z]*\d+)?/g, '');
-                    $(this).html(replace);
-                });
+        await page.goto("");
+        await page.evaluate(function () {
+            $('p').each(function () {
+                var replace = $(this).html().replace(/(?!1\.0)\d+\.\d+(\.\d+)?([\-a-z]*\d+)?/g, '');
+                $(this).html(replace);
             });
-        }, done);
+        });
+
+        expect(await page.screenshot({ fullPage: true })).to.matchImage('main_instance');
     });
 
-    it("should show the donation form when the update process is complete", function (done) {
-        expect.screenshot("updated").to.be.capture(function (page) {
-            page.click('.btn');
-        }, done);
+    it("should show the donation form when the update process is complete", async function() {
+        await page.click('.btn');
+        await page.waitForNetworkIdle();
+
+        expect(await page.screenshot({ fullPage: true })).to.matchImage('updated');
     });
 });
diff --git a/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterCode_httpUpdateSuccess.png b/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterCode_httpUpdateSuccess.png
index 1ef2d0d5e6..166f48bdd0 100644
--- a/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterCode_httpUpdateSuccess.png
+++ b/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterCode_httpUpdateSuccess.png
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:96bd09c46d200519ae891ff834497386b764b1cce5284e370e115623385881df
-size 240925
+oid sha256:d787baf820a6871341f7db215c08bbfcc3f223f1473a264fe409f23819509942
+size 224143
diff --git a/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterCode_httpsUpdateFail.png b/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterCode_httpsUpdateFail.png
index 272339640e..1e10ca08d1 100644
--- a/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterCode_httpsUpdateFail.png
+++ b/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterCode_httpsUpdateFail.png
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:304d1f7482f61e3ad6cb49c33b41f204afda7024d418fbab5cfc87fde5c0a7e7
-size 98350
+oid sha256:763715f2a99b3eed72a52b1b410a157a1e92abaa510cc0e0461273b92d91cf95
+size 98586
diff --git a/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterCode_newVersion.png b/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterCode_newVersion.png
index 01765d9d3f..11d95a9fe9 100644
--- a/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterCode_newVersion.png
+++ b/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterCode_newVersion.png
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e9345f029b8c3333ad2b1bd40ceac2be31185577872973d2e2a298f0d75f2fcd
-size 112434
+oid sha256:ea147473509fb34c67d23818e221f138b977305ae2a9c05234ff2e30e13adb81
+size 111947
diff --git a/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterDb_main.png b/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterDb_main.png
index 0806c0cb64..08f49a4f90 100644
--- a/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterDb_main.png
+++ b/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterDb_main.png
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c6d782438f1fb12fcc1e4ab18ae71eb431172faac14c3232d9133758952f1ae7
-size 306036
+oid sha256:126430076c074630db9191c1f9bfe4bd8fc8fb9ab687de5aa5afac3eb2b20cbb
+size 309789
diff --git a/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterDb_main_instance.png b/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterDb_main_instance.png
index da8551d607..10f13b39f8 100644
--- a/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterDb_main_instance.png
+++ b/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterDb_main_instance.png
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:600c26506ce387821e8df5aa82532aedd0fc43957389879d03d27bac3a25b63b
-size 307931
+oid sha256:f4bb48a56e3690905f04b97bc37898df39ae3073c5f8bcbd56d7f7f22ddf2cf8
+size 311679
diff --git a/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterDb_updated.png b/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterDb_updated.png
index 22592267a0..f6b16dcdc2 100644
--- a/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterDb_updated.png
+++ b/plugins/CoreUpdater/tests/UI/expected-screenshots/CoreUpdaterDb_updated.png
@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0d9ea8fcb6e99d4acb8be86eb8a2b21500321eec81544063397f79407208180e
-size 79160
+oid sha256:362dd259a2f9982befb3c470232654052aa1e0596c0b66b30c46a2f6efd85a1c
+size 77080