allow invalidating plugin archives only and archiving past data for plugins (#15889)

* Adding initial new code for cron archive rewrite.

* first pass at removing unused CronArchive code.

* unfinished commit

* fill out archiveinvalidator code

* getting some tests to pass

* unfinished commit

* fixing part of test

* Another test fix.

* another sql change

* fix broken merge or something else that went wrong

* Couple more fixes and extra logs.

* Fixing enough issues to get core archive command to run completely.

* Fix and log change.

* Fixed more segment/test related issues for CronArchiveTest. Includes optimization for no visits for period + segment process from handling.

* another optimization and possible build fix

* no visit optimization

* test fix

* Implement archiving_custom_ranges logic w/ queue based implementation

* fixes to get archivecrontest to work

* add logic to invalidate today period

* fix optimization and some tests

* Fixing more tests.

* Fixing more tests

* debug travis failure

* more test fixes

* more test fixes, removing more unneeded code, handling some TODOs

* Handle more TODOs including creating ArchiveFilter class for some cli options.

* tests and todos

* idarchives are specific to table + start on archivefilter tests

* one test

* more TODOs and tests

* more tests and todo taken care of

* handle more todos

* fixing more tests

* fix comment

* make sure autoarchiving is enabled for segments when cron archive picks them up

* Fixing test.

* apply more pr feedback

* order by date1 asc

* quick refactor

* use batch insert instead of createDummyArchives

* apply rest of pr feedback

* add removed events, add new test, fix an issue (when deleting idarchives older than do not lump all segments together).

* re-add fixed/shared siteids

* fix tests

* incomplete commit

* Insert archive entries into archive_invalidations table.

* Use invalidations table in core:archive and get ArchiveCronTest to pass.

* fixing some tests

* debugging travis

* fix more tests & remove DONE_IN_PROGRESS which is no longer used.

* fix more tests

* Allow forcing plugin specific archive in core:archive.

* When querying from archive data use all available archives including "all" archives and plugin specific archives.

* Adding some code for invalidating specific plugin archives.

* Get archive invalidation test to pass.

* add plugin capability to invalidate command

* Handle plugin only archives in core:archive.

* Add Archive test and get ArchiveCronTest to pass.

* update some expected files

* Fix some more tests.

* incomplete commit

* allow invalidating individual reports

* adding more API for DONE_PARTIAL support

* get archivecrontest to pass

* add archive processor tests

* fix some test randomnes

* when purging keep latest partial archives if there is no newer whole archive

* add rearchivereport method + some unfinished tests

* Add archiveReports API method, fix race condition in test, when archiving single report, always ignore inserting other reports.

* require archivers to handle partial archives themselves entirely instead of trying to do it automatically and allow requested report to be any string

* couple fixes

* Use core config option for last N montsh to invalidate.

* Add test for ArchiveSelector method.

* Ignore archives w/ deactivated plugins.

* Refactor queue looping into new QueueConsumer class.

* apply more review feedback + another fix

* invalidate segments too in reArchiveReport w/ etsts

* remove DONE_IN_PROGRESS, no longer used. use new status in query and add queue consumer test.

* forgot to add file

* delete old unneeded archives when finalizing a new one.

* tweak invalidation archive description

* add plugin archiving tests and get them to pass

* fix test

* many fixes

* fix another test

* update expected test files

* fix more tests

* last test fixes hopefully

* tweak log

* In case a column already exists, do not try to add it in an AddColumns migration or the entire migration will fail and no columns will be added.

* try to fix tests again

* fix again?

* apply some review feedback + fix test

* fix test

* fix another test

* couple fixes

* Remove extra param.

* apply pr feedback

* check for usable archive before invalidating and before initiating archiving

* fixing tests

* fixing tests

* fixing tests

* fix another test issue

* fix archiveinvalidator test

* fix one test and debug another

* more debugging

* fix test

* use twig

* remove no longer needed change

* add back previous logic

* fix tracking is not working

* apply pr feedback and add tests

* fixing tests

* update submodule

* debugging random travis failure

* update test

* more debugging

* more debugging

* another attempt at debugging

* Lets try this fix

* trying to fix the build

* debug

* simpler test

* fix test

* fix test

* fix test

* fix test

* fix test failure

* update screenshots

* update screenshots

Co-authored-by: Thomas Steur <tsteur@users.noreply.github.com>

9 files changed, 216 insertions, 93 deletions

diff --git a/plugins/CoreConsole/tests/System/ArchiveCronTest.php b/plugins/CoreConsole/tests/System/ArchiveCronTest.php
index e426778427..3b71dcd7ba 100644
--- a/plugins/CoreConsole/tests/System/ArchiveCronTest.php
+++ b/plugins/CoreConsole/tests/System/ArchiveCronTest.php
@@ -8,17 +8,21 @@
 namespace Piwik\Plugins\CoreConsole\tests\System;
 
 use Interop\Container\ContainerInterface;
+use Piwik\Archive\ArchiveInvalidator;
 use Piwik\Common;
 use Piwik\Config;
-use Piwik\CronArchive;
+use Piwik\Container\StaticContainer;
+use Piwik\DataAccess\ArchiveTableCreator;
 use Piwik\Date;
 use Piwik\Db;
 use Piwik\Option;
-use Piwik\Plugins\SitesManager\API;
+use Piwik\Segment;
+use Piwik\Sequence;
 use Piwik\Tests\Framework\TestCase\SystemTestCase;
 use Piwik\Tests\Fixtures\ManySitesImportedLogs;
 use Piwik\Tests\Framework\Fixture;
 use Exception;
+use Psr\Log\LoggerInterface;
 
 /**
  * Tests to call the cron core:archive command script and check there is no error,
@@ -36,6 +40,13 @@ class ArchiveCronTest extends SystemTestCase
      */
     public static $fixture = null; // initialized below class definition
 
+    public static function setUpBeforeClass(): void
+    {
+        parent::setUpBeforeClass();
+
+        Db::exec("UPDATE " . Common::prefixTable('site') . ' SET ts_created = \'2005-01-02 00:00:00\'');
+    }
+
     public function getApiForTesting()
     {
         $apiRequiringSegments = ['Goals.get', 'VisitFrequency.get'];
@@ -48,14 +59,26 @@ class ArchiveCronTest extends SystemTestCase
                                                           'periods'    => array('day', 'week', 'month', 'year'),
                                                           'segment'    => $info['definition'],
                                                           'testSuffix' => '_' . $segmentName));
-
-
         }
 
+        // ExamplePlugin metric
+        $results[] = ['ExamplePlugin.getExampleArchivedMetric', [
+            'idSite' => 'all',
+            'date' => '2007-04-05',
+            'periods' => ['day', 'week'],
+        ]];
+        $results[] = ['Actions.get', [
+            'idSite' => 'all',
+            'date' => '2007-04-05',
+            'periods' => ['day', 'week'],
+            'testSuffix' => '_examplePluginNoMetricsBecauseNoOtherPluginsArchived',
+        ]];
+
         // API Call Without segments
         $results[] = array('VisitsSummary.get', array('idSite'  => 'all',
                                                       'date'    => '2012-08-09',
                                                       'periods' => array('day', 'month', 'year',  'week')));
+
         $results[] = array($apiRequiringSegments, array('idSite'  => 'all',
             'date'    => '2012-08-09',
             'periods' => array('month')));
@@ -92,6 +115,18 @@ class ArchiveCronTest extends SystemTestCase
 
     public function testArchivePhpCron()
     {
+        // invalidate exampleplugin only archives in past
+        $invalidator = StaticContainer::get(ArchiveInvalidator::class);
+        $invalidator->markArchivesAsInvalidated([1], ['2007-04-05'], 'day', new Segment('', [1]), false, false, 'ExamplePlugin');
+
+        // track a visit in 2007-04-05 so it will archive (don't want to force archiving because then this test will take another 15 mins)
+        $tracker = Fixture::getTracker(1, '2007-04-05');
+        $tracker->setUrl('http://example.com/test/url');
+        Fixture::checkResponse($tracker->doTrackPageView('abcdefg'));
+
+        // empty the list so nothing is invalidated during core:archive (so we only archive ExamplePlugin and not all plugins)
+        $invalidator->forgetRememberedArchivedReportsToInvalidate(1, Date::factory('2007-04-05'));
+
         $output = $this->runArchivePhpCron();
 
         $expectedInvalidations = [];
@@ -118,6 +153,40 @@ class ArchiveCronTest extends SystemTestCase
         }
     }
 
+    /**
+     * @depends testArchivePhpCron
+     */
+    public function testArchivePhpCronWithSingleReportRearchive()
+    {
+        // invalidate a report so we get a partial archive (using the metric that gets incremented each time it is archived)
+        // (do it after the last run so we don't end up just re-using the ExamplePlugin archive)
+        $invalidator = StaticContainer::get(ArchiveInvalidator::class);
+        $invalidator->markArchivesAsInvalidated([1], ['2007-04-05'], 'day', new Segment('', [1]), false, false, 'ExamplePlugin.ExamplePlugin_example_metric2');
+
+        $sequence = new Sequence('ExamplePlugin_archiveCount');
+        $beforeCount = $sequence->getCurrentId();
+
+        $output = $this->runArchivePhpCron(['-vvv' => null]);
+
+        $afterCount = $sequence->getCurrentId();
+
+        $this->assertNotEquals($beforeCount, $afterCount, 'example plugin archiving was not triggered');
+
+        $this->runApiTests('ExamplePlugin.getExampleArchivedMetric', [
+            'idSite' => 'all',
+            'date' => '2007-04-05',
+            'periods' => ['day', 'week'],
+            'testSuffix' => '_singleMetric',
+        ]);
+
+        // test that latest archives for ExamplePlugin are partial
+        $archiveValues = Db::fetchAll("SELECT value FROM " . ArchiveTableCreator::getNumericTable(Date::factory('2007-04-05'))
+            . " WHERE `name` = 'done.ExamplePlugin' ORDER BY ts_archived DESC LIMIT 8");
+        $archiveValues = array_column($archiveValues, 'value');
+        $archiveValues = array_unique($archiveValues);
+        $this->assertEquals([5], $archiveValues);
+    }
+
     public function testArchivePhpCronArchivesFullRanges()
     {
         self::$fixture->getTestEnvironment()->overrideConfig('General', 'enable_browser_archiving_triggering', 0);
@@ -133,9 +202,11 @@ class ArchiveCronTest extends SystemTestCase
 
         $expectedInvalidations = [];
         $invalidationEntries = $this->getInvalidatedArchiveTableEntries();
+
         $invalidationEntries = array_filter($invalidationEntries, function ($entry) {
             return $entry['period'] == 5;
         });
+
         $this->assertEquals($expectedInvalidations, $invalidationEntries);
 
         $this->runApiTests(array(
@@ -143,8 +214,7 @@ class ArchiveCronTest extends SystemTestCase
             array('idSite'     => '1',
                 'date'       => '2012-08-09,2012-08-13',
                 'periods'    => array('range'),
-                'testSuffix' => '_range_archive'
-            )
+                'testSuffix' => '_range_archive')
         );
     }
 
diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest__ExamplePlugin.getExampleArchivedMetric_day.xml b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest__ExamplePlugin.getExampleArchivedMetric_day.xml
new file mode 100644
index 0000000000..2e1a657b1f
--- /dev/null
+++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest__ExamplePlugin.getExampleArchivedMetric_day.xml
@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<results>
+	<result idSite="1">
+		<ExamplePlugin_example_metric>3382</ExamplePlugin_example_metric>
+		<ExamplePlugin_example_metric2>51</ExamplePlugin_example_metric2>
+	</result>
+	<result idSite="2" />
+	<result idSite="3" />
+</results>
\ No newline at end of file
diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_archive_php_cron_output.txt b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_archive_php_cron_output.txt
index e8b09e0083..81032bb1a7 100644
--- a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_archive_php_cron_output.txt
+++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_archive_php_cron_output.txt
@@ -10,90 +10,86 @@ try 'php archive.php --url=http://your.piwik/path'
 -------------------------------------------------------
 
 
-INFO [2020-06-25 09:52:47] 10351  ---------------------------
-INFO [2020-06-25 09:52:47] 10351  INIT
-INFO [2020-06-25 09:52:47] 10351  Running Matomo 4.0.0-b2 as Super User
-INFO [2020-06-25 09:52:47] 10351  ---------------------------
-INFO [2020-06-25 09:52:47] 10351  NOTES
-INFO [2020-06-25 09:52:47] 10351  - If you execute this script at least once per hour (or more often) in a crontab, you may disable 'Browser trigger archiving' in Matomo UI > Settings > General Settings.
-INFO [2020-06-25 09:52:47] 10351    See the doc at: https://matomo.org/docs/setup-auto-archiving/
-INFO [2020-06-25 09:52:47] 10351  - Async process archiving supported, using CliMulti.
-INFO [2020-06-25 09:52:47] 10351  - Reports for today will be processed at most every 900 seconds. You can change this value in Matomo UI > Settings > General Settings.
-INFO [2020-06-25 09:52:47] 10351  ---------------------------
-INFO [2020-06-25 09:52:47] 10351  START
-INFO [2020-06-25 09:52:47] 10351  Starting Matomo reports archiving...
-INFO [2020-06-25 09:52:47] 10351  Checking for queued invalidations...
-INFO [2020-06-25 09:52:47] 10351    Will invalidate archived reports for 2014-03-13 for following websites ids: 3,1
-INFO [2020-06-25 09:52:47] 10351    Will invalidate archived reports for 2014-03-12 for following websites ids: 1,3
-INFO [2020-06-25 09:52:47] 10351    Will invalidate archived reports for 2012-09-30 for following websites ids: 1
-INFO [2020-06-25 09:52:47] 10351    Will invalidate archived reports for 2012-08-19 for following websites ids: 1
-INFO [2020-06-25 09:52:47] 10351    Will invalidate archived reports for 2012-08-15 for following websites ids: 1,3,2
-INFO [2020-06-25 09:52:47] 10351    Will invalidate archived reports for 2012-08-11 for following websites ids: 1
-INFO [2020-06-25 09:52:47] 10351    Will invalidate archived reports for 2012-08-10 for following websites ids: 1
-INFO [2020-06-25 09:52:47] 10351    Will invalidate archived reports for 2012-08-09 for following websites ids: 1
-INFO [2020-06-25 09:52:47] 10351    Segment "browserCode==IE" was created or changed recently and will therefore archive today (for site ID = 1)
-INFO [2020-06-25 09:52:47] 10351    Segment "visitCount<=5;visitorType!=non-existing-type;daysSinceFirstVisit<=50" was created or changed recently and will therefore archive today (for site ID = 1)
-INFO [2020-06-25 09:52:47] 10351    Segment "visitCount<=5;visitorType!=re%2C%3Btest%20is%20encoded;daysSinceFirstVisit<=50" was created or changed recently and will therefore archive today (for site ID = 1)
-INFO [2020-06-25 09:52:47] 10351  Done invalidating
-INFO [2020-06-25 09:52:47] 10351  Start processing archives for site 1.
-INFO [2020-06-25 09:52:50] 10351  Archived website id 1, period = day, date = 2014-03-13, segment = '', 11 visits found. Time elapsed: 1.477s
-INFO [2020-06-25 09:52:50] 10351  Archived website id 1, period = day, date = 2014-03-12, segment = '', 1 visits found. Time elapsed: 1.477s
-INFO [2020-06-25 09:52:50] 10351  Archived website id 1, period = day, date = 2012-09-30, segment = '', 1 visits found. Time elapsed: 1.477s
-INFO [2020-06-25 09:52:51] 10351  Archived website id 1, period = week, date = 2014-03-10, segment = '', 12 visits found. Time elapsed: 1.116s
-INFO [2020-06-25 09:52:51] 10351  Archived website id 1, period = week, date = 2012-09-24, segment = '', 1 visits found. Time elapsed: 1.151s
-INFO [2020-06-25 09:52:51] 10351  Archived website id 1, period = week, date = 2012-08-13, segment = '', 3 visits found. Time elapsed: 1.425s
-INFO [2020-06-25 09:52:54] 10351  Archived website id 1, period = month, date = 2014-03-01, segment = '', 12 visits found. Time elapsed: 1.115s
-INFO [2020-06-25 09:52:54] 10351  Archived website id 1, period = month, date = 2012-09-01, segment = '', 1 visits found. Time elapsed: 1.115s
-INFO [2020-06-25 09:52:54] 10351  Archived website id 1, period = month, date = 2012-08-01, segment = '', 33 visits found. Time elapsed: 1.936s
-INFO [2020-06-25 09:52:55] 10351  Archived website id 1, period = year, date = 2014-01-01, segment = '', 12 visits found. Time elapsed: 0.919s
-INFO [2020-06-25 09:52:55] 10351  Archived website id 1, period = year, date = 2012-01-01, segment = '', 34 visits found. Time elapsed: 0.919s
-INFO [2020-06-25 09:52:59] 10351  Archived website id 1, period = year, date = 2014-01-01, segment = 'browserCode%3D%3DIE', 1 visits found. Time elapsed: 3.088s
-INFO [2020-06-25 09:52:59] 10351  Archived website id 1, period = year, date = 2014-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 12 visits found. Time elapsed: 3.967s
-INFO [2020-06-25 09:52:59] 10351  Archived website id 1, period = year, date = 2014-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 12 visits found. Time elapsed: 3.967s
-INFO [2020-06-25 09:53:01] 10351  Archived website id 1, period = month, date = 2012-09-01, segment = 'browserCode%3D%3DIE', 0 visits found. Time elapsed: 1.113s
-INFO [2020-06-25 09:53:01] 10351  Archived website id 1, period = month, date = 2012-09-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 1 visits found. Time elapsed: 1.996s
-INFO [2020-06-25 09:53:01] 10351  Archived website id 1, period = month, date = 2012-09-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 1 visits found. Time elapsed: 1.996s
-INFO [2020-06-25 09:53:04] 10351  Archived website id 1, period = week, date = 2012-08-13, segment = 'browserCode%3D%3DIE', 1 visits found. Time elapsed: 1.807s
-INFO [2020-06-25 09:53:04] 10351  Archived website id 1, period = week, date = 2012-08-13, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 3 visits found. Time elapsed: 1.859s
-INFO [2020-06-25 09:53:04] 10351  Archived website id 1, period = week, date = 2012-08-13, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 3 visits found. Time elapsed: 2.129s
-INFO [2020-06-25 09:53:05] 10351  Archived website id 1, period = day, date = 2012-08-11, segment = 'browserCode%3D%3DIE', 1 visits found. Time elapsed: 1.130s
-INFO [2020-06-25 09:53:05] 10351  Archived website id 1, period = day, date = 2012-08-11, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 11 visits found. Time elapsed: 1.130s
-INFO [2020-06-25 09:53:05] 10351  Archived website id 1, period = day, date = 2012-08-11, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 11 visits found. Time elapsed: 1.403s
-INFO [2020-06-25 09:53:06] 10351  Archived website id 1, period = day, date = 2012-08-10, segment = 'browserCode%3D%3DIE', 3 visits found. Time elapsed: 1.114s
-INFO [2020-06-25 09:53:06] 10351  Archived website id 1, period = day, date = 2012-08-10, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 8 visits found. Time elapsed: 1.426s
-INFO [2020-06-25 09:53:06] 10351  Archived website id 1, period = day, date = 2012-08-10, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 8 visits found. Time elapsed: 1.426s
-INFO [2020-06-25 09:53:08] 10351  Archived website id 1, period = day, date = 2012-08-09, segment = 'browserCode%3D%3DIE', 1 visits found. Time elapsed: 1.099s
-INFO [2020-06-25 09:53:08] 10351  Archived website id 1, period = day, date = 2012-08-09, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 11 visits found. Time elapsed: 1.417s
-INFO [2020-06-25 09:53:08] 10351  Archived website id 1, period = day, date = 2012-08-09, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 11 visits found. Time elapsed: 1.417s
-INFO [2020-06-25 09:53:10] 10351  Archived website id 1, period = week, date = 2012-08-06, segment = 'browserCode%3D%3DIE', 5 visits found. Time elapsed: 1.442s
-INFO [2020-06-25 09:53:10] 10351  Archived website id 1, period = week, date = 2012-08-06, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 30 visits found. Time elapsed: 1.442s
-INFO [2020-06-25 09:53:10] 10351  Archived website id 1, period = week, date = 2012-08-06, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 30 visits found. Time elapsed: 1.714s
-INFO [2020-06-25 09:53:11] 10351  Archived website id 1, period = month, date = 2012-08-01, segment = 'browserCode%3D%3DIE', 6 visits found. Time elapsed: 1.403s
-INFO [2020-06-25 09:53:11] 10351  Archived website id 1, period = month, date = 2012-08-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 33 visits found. Time elapsed: 1.403s
-INFO [2020-06-25 09:53:11] 10351  Archived website id 1, period = month, date = 2012-08-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 33 visits found. Time elapsed: 1.676s
-INFO [2020-06-25 09:53:13] 10351  Archived website id 1, period = year, date = 2012-01-01, segment = 'browserCode%3D%3DIE', 6 visits found. Time elapsed: 1.421s
-INFO [2020-06-25 09:53:13] 10351  Archived website id 1, period = year, date = 2012-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 34 visits found. Time elapsed: 1.728s
-INFO [2020-06-25 09:53:13] 10351  Archived website id 1, period = year, date = 2012-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 34 visits found. Time elapsed: 1.728s
-INFO [2020-06-25 09:53:13] 10351  Finished archiving for site 1, 38 API requests, Time elapsed: 25.964s [1 / 3 done]
-INFO [2020-06-25 09:53:13] 10351  Start processing archives for site 2.
-INFO [2020-06-25 09:53:14] 10351  Archived website id 2, period = day, date = 2012-08-15, segment = '', 1 visits found. Time elapsed: 0.556s
-INFO [2020-06-25 09:53:14] 10351  Archived website id 2, period = week, date = 2012-08-13, segment = '', 1 visits found. Time elapsed: 0.555s
-INFO [2020-06-25 09:53:15] 10351  Archived website id 2, period = month, date = 2012-08-01, segment = '', 1 visits found. Time elapsed: 0.555s
-INFO [2020-06-25 09:53:16] 10351  Archived website id 2, period = year, date = 2012-01-01, segment = '', 1 visits found. Time elapsed: 0.554s
-INFO [2020-06-25 09:53:16] 10351  Finished archiving for site 2, 4 API requests, Time elapsed: 2.374s [2 / 3 done]
-INFO [2020-06-25 09:53:16] 10351  Start processing archives for site 3.
-INFO [2020-06-25 09:53:17] 10351  Archived website id 3, period = day, date = 2014-03-13, segment = '', 11 visits found. Time elapsed: 1.116s
-INFO [2020-06-25 09:53:17] 10351  Archived website id 3, period = day, date = 2014-03-12, segment = '', 1 visits found. Time elapsed: 1.116s
-INFO [2020-06-25 09:53:17] 10351  Archived website id 3, period = day, date = 2012-08-15, segment = '', 3 visits found. Time elapsed: 1.116s
-INFO [2020-06-25 09:53:18] 10351  Archived website id 3, period = week, date = 2014-03-10, segment = '', 12 visits found. Time elapsed: 0.930s
-INFO [2020-06-25 09:53:18] 10351  Archived website id 3, period = week, date = 2012-08-13, segment = '', 3 visits found. Time elapsed: 0.930s
-INFO [2020-06-25 09:53:19] 10351  Archived website id 3, period = month, date = 2014-03-01, segment = '', 12 visits found. Time elapsed: 0.932s
-INFO [2020-06-25 09:53:19] 10351  Archived website id 3, period = month, date = 2012-08-01, segment = '', 3 visits found. Time elapsed: 0.932s
-INFO [2020-06-25 09:53:20] 10351  Archived website id 3, period = year, date = 2014-01-01, segment = '', 12 visits found. Time elapsed: 0.923s
-INFO [2020-06-25 09:53:20] 10351  Archived website id 3, period = year, date = 2012-01-01, segment = '', 3 visits found. Time elapsed: 0.923s
-INFO [2020-06-25 09:53:20] 10351  Finished archiving for site 3, 9 API requests, Time elapsed: 4.088s [3 / 3 done]
-INFO [2020-06-25 09:53:20] 10351  ---------------------------
-INFO [2020-06-25 09:53:20] 10351  SCHEDULED TASKS
-INFO [2020-06-25 09:53:20] 10351  Starting Scheduled tasks...
-INFO [2020-06-25 09:53:20] 10351  done
-INFO [2020-06-25 09:53:20] 10351  ---------------------------
\ No newline at end of file
+INFO [2020-07-06 21:59:34] 10307  ---------------------------
+INFO [2020-07-06 21:59:34] 10307  INIT
+INFO [2020-07-06 21:59:34] 10307  Running Matomo 4.0.0-b2 as Super User
+INFO [2020-07-06 21:59:34] 10307  ---------------------------
+INFO [2020-07-06 21:59:34] 10307  NOTES
+INFO [2020-07-06 21:59:34] 10307  - If you execute this script at least once per hour (or more often) in a crontab, you may disable 'Browser trigger archiving' in Matomo UI > Settings > General Settings.
+INFO [2020-07-06 21:59:34] 10307    See the doc at: https://matomo.org/docs/setup-auto-archiving/
+INFO [2020-07-06 21:59:34] 10307  - Async process archiving supported, using CliMulti.
+INFO [2020-07-06 21:59:34] 10307  - Reports for today will be processed at most every 900 seconds. You can change this value in Matomo UI > Settings > General Settings.
+INFO [2020-07-06 21:59:34] 10307  ---------------------------
+INFO [2020-07-06 21:59:34] 10307  START
+INFO [2020-07-06 21:59:34] 10307  Starting Matomo reports archiving...
+INFO [2020-07-06 21:59:34] 10307  Checking for queued invalidations...
+INFO [2020-07-06 21:59:34] 10307    Will invalidate archived reports for 2014-03-13 for following websites ids: 1,3
+INFO [2020-07-06 21:59:34] 10307    Will invalidate archived reports for 2014-03-12 for following websites ids: 3,1
+INFO [2020-07-06 21:59:34] 10307    Will invalidate archived reports for 2012-09-30 for following websites ids: 1
+INFO [2020-07-06 21:59:34] 10307    Will invalidate archived reports for 2012-08-19 for following websites ids: 1
+INFO [2020-07-06 21:59:34] 10307    Will invalidate archived reports for 2012-08-15 for following websites ids: 2,1,3
+INFO [2020-07-06 21:59:34] 10307    Will invalidate archived reports for 2012-08-11 for following websites ids: 1
+INFO [2020-07-06 21:59:34] 10307    Will invalidate archived reports for 2012-08-10 for following websites ids: 1
+INFO [2020-07-06 21:59:34] 10307    Will invalidate archived reports for 2012-08-09 for following websites ids: 1
+INFO [2020-07-06 21:59:34] 10307    Segment "browserCode==IE" was created or changed recently and will therefore archive today (for site ID = 1)
+INFO [2020-07-06 21:59:34] 10307    Segment "visitCount<=5;visitorType!=non-existing-type;daysSinceFirstVisit<=50" was created or changed recently and will therefore archive today (for site ID = 1)
+INFO [2020-07-06 21:59:34] 10307    Segment "visitCount<=5;visitorType!=re%2C%3Btest%20is%20encoded;daysSinceFirstVisit<=50" was created or changed recently and will therefore archive today (for site ID = 1)
+INFO [2020-07-06 21:59:34] 10307  Done invalidating
+INFO [2020-07-06 21:59:34] 10307  Start processing archives for site 1.
+INFO [2020-07-06 21:59:37] 10307  Archived website id 1, period = day, date = 2014-03-13, segment = '', 11 visits found. Time elapsed: 1.124s
+INFO [2020-07-06 21:59:37] 10307  Archived website id 1, period = day, date = 2014-03-12, segment = '', 1 visits found. Time elapsed: 1.124s
+INFO [2020-07-06 21:59:37] 10307  Archived website id 1, period = day, date = 2012-09-30, segment = '', 1 visits found. Time elapsed: 1.124s
+INFO [2020-07-06 21:59:38] 10307  Archived website id 1, period = week, date = 2014-03-10, segment = '', 12 visits found. Time elapsed: 1.761s
+INFO [2020-07-06 21:59:38] 10307  Archived website id 1, period = week, date = 2012-09-24, segment = '', 1 visits found. Time elapsed: 1.761s
+INFO [2020-07-06 21:59:38] 10307  Archived website id 1, period = day, date = 2012-08-19, segment = '', 1 visits found. Time elapsed: 1.761s
+INFO [2020-07-06 21:59:40] 10307  Archived website id 1, period = month, date = 2014-03-01, segment = '', 12 visits found. Time elapsed: 1.743s
+INFO [2020-07-06 21:59:40] 10307  Archived website id 1, period = month, date = 2012-09-01, segment = '', 1 visits found. Time elapsed: 1.743s
+INFO [2020-07-06 21:59:40] 10307  Archived website id 1, period = day, date = 2012-08-15, segment = '', 2 visits found. Time elapsed: 1.743s
+INFO [2020-07-06 21:59:43] 10307  Archived website id 1, period = year, date = 2014-01-01, segment = '', 12 visits found. Time elapsed: 2.510s
+INFO [2020-07-06 21:59:43] 10307  Archived website id 1, period = week, date = 2012-08-13, segment = '', 3 visits found. Time elapsed: 2.510s
+INFO [2020-07-06 21:59:43] 10307  Archived website id 1, period = day, date = 2012-08-11, segment = '', 11 visits found. Time elapsed: 2.510s
+INFO [2020-07-06 21:59:47] 10307  Archived website id 1, period = year, date = 2014-01-01, segment = 'browserCode%3D%3DIE', 1 visits found. Time elapsed: 3.015s
+INFO [2020-07-06 21:59:47] 10307  Archived website id 1, period = year, date = 2012-01-01, segment = 'browserCode%3D%3DIE', 6 visits found. Time elapsed: 4.118s
+INFO [2020-07-06 21:59:47] 10307  Archived website id 1, period = day, date = 2007-04-05, segment = '', plugin = ExamplePlugin, 1 visits found. Time elapsed: 4.118s
+INFO [2020-07-06 21:59:53] 10307  Archived website id 1, period = year, date = 2014-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 12 visits found. Time elapsed: 4.601s
+INFO [2020-07-06 21:59:53] 10307  Archived website id 1, period = year, date = 2012-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 34 visits found. Time elapsed: 5.707s
+INFO [2020-07-06 21:59:53] 10307  Archived website id 1, period = week, date = 2007-04-02, segment = '', plugin = ExamplePlugin, 1 visits found. Time elapsed: 5.707s
+INFO [2020-07-06 21:59:59] 10307  Archived website id 1, period = year, date = 2014-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 12 visits found. Time elapsed: 3.018s
+INFO [2020-07-06 21:59:59] 10307  Archived website id 1, period = year, date = 2012-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 34 visits found. Time elapsed: 5.246s
+INFO [2020-07-06 21:59:59] 10307  Archived website id 1, period = month, date = 2007-04-01, segment = '', plugin = ExamplePlugin, 1 visits found. Time elapsed: 5.246s
+INFO [2020-07-06 22:00:00] 10307  Archived website id 1, period = day, date = 2012-08-10, segment = '', 8 visits found. Time elapsed: 0.632s
+INFO [2020-07-06 22:00:00] 10307  Archived website id 1, period = day, date = 2012-08-09, segment = '', 11 visits found. Time elapsed: 0.916s
+INFO [2020-07-06 22:00:01] 10307  Archived website id 1, period = week, date = 2012-08-06, segment = '', 30 visits found. Time elapsed: 0.827s
+INFO [2020-07-06 22:00:02] 10307  Archived website id 1, period = month, date = 2012-08-01, segment = '', 33 visits found. Time elapsed: 1.104s
+INFO [2020-07-06 22:00:03] 10307  Archived website id 1, period = year, date = 2012-01-01, segment = '', 34 visits found. Time elapsed: 1.102s
+INFO [2020-07-06 22:00:03] 10307  Archived website id 1, period = year, date = 2007-01-01, segment = '', plugin = ExamplePlugin, 1 visits found. Time elapsed: 0.564s
+INFO [2020-07-06 22:00:03] 10307  Finished archiving for site 1, 27 API requests, Time elapsed: 28.981s [1 / 3 done]
+INFO [2020-07-06 22:00:04] 10307  Start processing archives for site 2.
+INFO [2020-07-06 22:00:04] 10307  Archived website id 2, period = day, date = 2012-08-15, segment = '', 1 visits found. Time elapsed: 0.549s
+INFO [2020-07-06 22:00:05] 10307  Archived website id 2, period = week, date = 2012-08-13, segment = '', 1 visits found. Time elapsed: 0.826s
+INFO [2020-07-06 22:00:06] 10307  Archived website id 2, period = month, date = 2012-08-01, segment = '', 1 visits found. Time elapsed: 0.825s
+INFO [2020-07-06 22:00:07] 10307  Archived website id 2, period = year, date = 2012-01-01, segment = '', 1 visits found. Time elapsed: 1.110s
+INFO [2020-07-06 22:00:07] 10307  Finished archiving for site 2, 4 API requests, Time elapsed: 3.469s [2 / 3 done]
+INFO [2020-07-06 22:00:07] 10307  Start processing archives for site 3.
+INFO [2020-07-06 22:00:08] 10307  Archived website id 3, period = day, date = 2014-03-13, segment = '', 11 visits found. Time elapsed: 0.834s
+INFO [2020-07-06 22:00:08] 10307  Archived website id 3, period = day, date = 2014-03-12, segment = '', 1 visits found. Time elapsed: 0.834s
+INFO [2020-07-06 22:00:08] 10307  Archived website id 3, period = day, date = 2012-08-15, segment = '', 3 visits found. Time elapsed: 1.115s
+INFO [2020-07-06 22:00:09] 10307  Archived website id 3, period = week, date = 2014-03-10, segment = '', 12 visits found. Time elapsed: 1.257s
+INFO [2020-07-06 22:00:09] 10307  Archived website id 3, period = week, date = 2012-08-13, segment = '', 3 visits found. Time elapsed: 1.257s
+INFO [2020-07-06 22:00:11] 10307  Archived website id 3, period = month, date = 2014-03-01, segment = '', 12 visits found. Time elapsed: 1.266s
+INFO [2020-07-06 22:00:11] 10307  Archived website id 3, period = month, date = 2012-08-01, segment = '', 3 visits found. Time elapsed: 1.818s
+INFO [2020-07-06 22:00:13] 10307  Archived website id 3, period = year, date = 2014-01-01, segment = '', 12 visits found. Time elapsed: 1.899s
+INFO [2020-07-06 22:00:13] 10307  Archived website id 3, period = year, date = 2012-01-01, segment = '', 3 visits found. Time elapsed: 1.920s
+INFO [2020-07-06 22:00:13] 10307  Finished archiving for site 3, 9 API requests, Time elapsed: 6.303s [3 / 3 done]
+INFO [2020-07-06 22:00:13] 10307  Done archiving!
+INFO [2020-07-06 22:00:13] 10307  ---------------------------
+INFO [2020-07-06 22:00:13] 10307  SUMMARY
+INFO [2020-07-06 22:00:13] 10307  Processed 40 archives.
+INFO [2020-07-06 22:00:13] 10307  Total API requests: 40
+INFO [2020-07-06 22:00:13] 10307  done: 40 req, 39477 ms, no error
+INFO [2020-07-06 22:00:13] 10307  Time elapsed: 39.477s
+INFO [2020-07-06 22:00:13] 10307  ---------------------------
+INFO [2020-07-06 22:00:13] 10307  SCHEDULED TASKS
+INFO [2020-07-06 22:00:13] 10307  Starting Scheduled tasks...
+INFO [2020-07-06 22:00:13] 10307  done
+INFO [2020-07-06 22:00:13] 10307  ---------------------------
\ No newline at end of file
diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_examplePluginNoMetricsBecauseNoOtherPluginsArchived_noOptions__Actions.get_day.xml b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_examplePluginNoMetricsBecauseNoOtherPluginsArchived_noOptions__Actions.get_day.xml
new file mode 100644
index 0000000000..dd52dc4ff4
--- /dev/null
+++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_examplePluginNoMetricsBecauseNoOtherPluginsArchived_noOptions__Actions.get_day.xml
@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<results>
+	<result idSite="1" />
+	<result idSite="2" />
+	<result idSite="3" />
+</results>
\ No newline at end of file
diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_examplePluginNoMetricsBecauseNoOtherPluginsArchived_noOptions__Actions.get_week.xml b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_examplePluginNoMetricsBecauseNoOtherPluginsArchived_noOptions__Actions.get_week.xml
new file mode 100644
index 0000000000..dd52dc4ff4
--- /dev/null
+++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_examplePluginNoMetricsBecauseNoOtherPluginsArchived_noOptions__Actions.get_week.xml
@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<results>
+	<result idSite="1" />
+	<result idSite="2" />
+	<result idSite="3" />
+</results>
\ No newline at end of file
diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_noOptions__ExamplePlugin.getExampleArchivedMetric_day.xml b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_noOptions__ExamplePlugin.getExampleArchivedMetric_day.xml
new file mode 100644
index 0000000000..bbba10abb2
--- /dev/null
+++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_noOptions__ExamplePlugin.getExampleArchivedMetric_day.xml
@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<results>
+	<result idSite="1">
+		<ExamplePlugin_example_metric>3382</ExamplePlugin_example_metric>
+		<ExamplePlugin_example_metric2>60</ExamplePlugin_example_metric2>
+	</result>
+	<result idSite="2" />
+	<result idSite="3" />
+</results>
\ No newline at end of file
diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_noOptions__ExamplePlugin.getExampleArchivedMetric_week.xml b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_noOptions__ExamplePlugin.getExampleArchivedMetric_week.xml
new file mode 100644
index 0000000000..bbba10abb2
--- /dev/null
+++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_noOptions__ExamplePlugin.getExampleArchivedMetric_week.xml
@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<results>
+	<result idSite="1">
+		<ExamplePlugin_example_metric>3382</ExamplePlugin_example_metric>
+		<ExamplePlugin_example_metric2>60</ExamplePlugin_example_metric2>
+	</result>
+	<result idSite="2" />
+	<result idSite="3" />
+</results>
\ No newline at end of file
diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_singleMetric__ExamplePlugin.getExampleArchivedMetric_day.xml b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_singleMetric__ExamplePlugin.getExampleArchivedMetric_day.xml
new file mode 100644
index 0000000000..4214cca45d
--- /dev/null
+++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_singleMetric__ExamplePlugin.getExampleArchivedMetric_day.xml
@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<results>
+	<result idSite="1">
+		<ExamplePlugin_example_metric>3382</ExamplePlugin_example_metric>
+		<ExamplePlugin_example_metric2>90</ExamplePlugin_example_metric2>
+	</result>
+	<result idSite="2" />
+	<result idSite="3" />
+</results>
\ No newline at end of file
diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_singleMetric__ExamplePlugin.getExampleArchivedMetric_week.xml b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_singleMetric__ExamplePlugin.getExampleArchivedMetric_week.xml
new file mode 100644
index 0000000000..4214cca45d
--- /dev/null
+++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_singleMetric__ExamplePlugin.getExampleArchivedMetric_week.xml
@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="utf-8" ?>
+<results>
+	<result idSite="1">
+		<ExamplePlugin_example_metric>3382</ExamplePlugin_example_metric>
+		<ExamplePlugin_example_metric2>90</ExamplePlugin_example_metric2>
+	</result>
+	<result idSite="2" />
+	<result idSite="3" />
+</results>
\ No newline at end of file