diff options
author | diosmosis <diosmosis@users.noreply.github.com> | 2020-08-04 05:59:58 +0300 |
---|---|---|
committer | GitHub <noreply@github.com> | 2020-08-04 05:59:58 +0300 |
commit | f5e9420a987340b036fa342e876ab92e314f4ec7 (patch) | |
tree | 2267e9eafe8b6577c4f7d3219d39c284a90677c6 /plugins/CoreConsole | |
parent | 2394c8c954d46d1ca9fe8d1304405e7fd6727c89 (diff) |
allow invalidating plugin archives only and archiving past data for plugins (#15889)
* Adding initial new code for cron archive rewrite.
* first pass at removing unused CronArchive code.
* unfinished commit
* fill out archiveinvalidator code
* getting some tests to pass
* unfinished commit
* fixing part of test
* Another test fix.
* another sql change
* fix broken merge or something else that went wrong
* Couple more fixes and extra logs.
* Fixing enough issues to get core archive command to run completely.
* Fix and log change.
* Fixed more segment/test related issues for CronArchiveTest. Includes optimization for no visits for period + segment process from handling.
* another optimization and possible build fix
* no visit optimization
* test fix
* Implement archiving_custom_ranges logic w/ queue based implementation
* fixes to get archivecrontest to work
* add logic to invalidate today period
* fix optimization and some tests
* Fixing more tests.
* Fixing more tests
* debug travis failure
* more test fixes
* more test fixes, removing more unneeded code, handling some TODOs
* Handle more TODOs including creating ArchiveFilter class for some cli options.
* tests and todos
* idarchives are specific to table + start on archivefilter tests
* one test
* more TODOs and tests
* more tests and todo taken care of
* handle more todos
* fixing more tests
* fix comment
* make sure autoarchiving is enabled for segments when cron archive picks them up
* Fixing test.
* apply more pr feedback
* order by date1 asc
* quick refactor
* use batch insert instead of createDummyArchives
* apply rest of pr feedback
* add removed events, add new test, fix an issue (when deleting idarchives older than do not lump all segments together).
* re-add fixed/shared siteids
* fix tests
* incomplete commit
* Insert archive entries into archive_invalidations table.
* Use invalidations table in core:archive and get ArchiveCronTest to pass.
* fixing some tests
* debugging travis
* fix more tests & remove DONE_IN_PROGRESS which is no longer used.
* fix more tests
* Allow forcing plugin specific archive in core:archive.
* When querying from archive data use all available archives including "all" archives and plugin specific archives.
* Adding some code for invalidating specific plugin archives.
* Get archive invalidation test to pass.
* add plugin capability to invalidate command
* Handle plugin only archives in core:archive.
* Add Archive test and get ArchiveCronTest to pass.
* update some expected files
* Fix some more tests.
* incomplete commit
* allow invalidating individual reports
* adding more API for DONE_PARTIAL support
* get archivecrontest to pass
* add archive processor tests
* fix some test randomnes
* when purging keep latest partial archives if there is no newer whole archive
* add rearchivereport method + some unfinished tests
* Add archiveReports API method, fix race condition in test, when archiving single report, always ignore inserting other reports.
* require archivers to handle partial archives themselves entirely instead of trying to do it automatically and allow requested report to be any string
* couple fixes
* Use core config option for last N montsh to invalidate.
* Add test for ArchiveSelector method.
* Ignore archives w/ deactivated plugins.
* Refactor queue looping into new QueueConsumer class.
* apply more review feedback + another fix
* invalidate segments too in reArchiveReport w/ etsts
* remove DONE_IN_PROGRESS, no longer used. use new status in query and add queue consumer test.
* forgot to add file
* delete old unneeded archives when finalizing a new one.
* tweak invalidation archive description
* add plugin archiving tests and get them to pass
* fix test
* many fixes
* fix another test
* update expected test files
* fix more tests
* last test fixes hopefully
* tweak log
* In case a column already exists, do not try to add it in an AddColumns migration or the entire migration will fail and no columns will be added.
* try to fix tests again
* fix again?
* apply some review feedback + fix test
* fix test
* fix another test
* couple fixes
* Remove extra param.
* apply pr feedback
* check for usable archive before invalidating and before initiating archiving
* fixing tests
* fixing tests
* fixing tests
* fix another test issue
* fix archiveinvalidator test
* fix one test and debug another
* more debugging
* fix test
* use twig
* remove no longer needed change
* add back previous logic
* fix tracking is not working
* apply pr feedback and add tests
* fixing tests
* update submodule
* debugging random travis failure
* update test
* more debugging
* more debugging
* another attempt at debugging
* Lets try this fix
* trying to fix the build
* debug
* simpler test
* fix test
* fix test
* fix test
* fix test
* fix test failure
* update screenshots
* update screenshots
Co-authored-by: Thomas Steur <tsteur@users.noreply.github.com>
Diffstat (limited to 'plugins/CoreConsole')
9 files changed, 216 insertions, 93 deletions
diff --git a/plugins/CoreConsole/tests/System/ArchiveCronTest.php b/plugins/CoreConsole/tests/System/ArchiveCronTest.php index e426778427..3b71dcd7ba 100644 --- a/plugins/CoreConsole/tests/System/ArchiveCronTest.php +++ b/plugins/CoreConsole/tests/System/ArchiveCronTest.php @@ -8,17 +8,21 @@ namespace Piwik\Plugins\CoreConsole\tests\System; use Interop\Container\ContainerInterface; +use Piwik\Archive\ArchiveInvalidator; use Piwik\Common; use Piwik\Config; -use Piwik\CronArchive; +use Piwik\Container\StaticContainer; +use Piwik\DataAccess\ArchiveTableCreator; use Piwik\Date; use Piwik\Db; use Piwik\Option; -use Piwik\Plugins\SitesManager\API; +use Piwik\Segment; +use Piwik\Sequence; use Piwik\Tests\Framework\TestCase\SystemTestCase; use Piwik\Tests\Fixtures\ManySitesImportedLogs; use Piwik\Tests\Framework\Fixture; use Exception; +use Psr\Log\LoggerInterface; /** * Tests to call the cron core:archive command script and check there is no error, @@ -36,6 +40,13 @@ class ArchiveCronTest extends SystemTestCase */ public static $fixture = null; // initialized below class definition + public static function setUpBeforeClass(): void + { + parent::setUpBeforeClass(); + + Db::exec("UPDATE " . Common::prefixTable('site') . ' SET ts_created = \'2005-01-02 00:00:00\''); + } + public function getApiForTesting() { $apiRequiringSegments = ['Goals.get', 'VisitFrequency.get']; @@ -48,14 +59,26 @@ class ArchiveCronTest extends SystemTestCase 'periods' => array('day', 'week', 'month', 'year'), 'segment' => $info['definition'], 'testSuffix' => '_' . $segmentName)); - - } + // ExamplePlugin metric + $results[] = ['ExamplePlugin.getExampleArchivedMetric', [ + 'idSite' => 'all', + 'date' => '2007-04-05', + 'periods' => ['day', 'week'], + ]]; + $results[] = ['Actions.get', [ + 'idSite' => 'all', + 'date' => '2007-04-05', + 'periods' => ['day', 'week'], + 'testSuffix' => '_examplePluginNoMetricsBecauseNoOtherPluginsArchived', + ]]; + // API Call Without segments $results[] = array('VisitsSummary.get', array('idSite' => 'all', 'date' => '2012-08-09', 'periods' => array('day', 'month', 'year', 'week'))); + $results[] = array($apiRequiringSegments, array('idSite' => 'all', 'date' => '2012-08-09', 'periods' => array('month'))); @@ -92,6 +115,18 @@ class ArchiveCronTest extends SystemTestCase public function testArchivePhpCron() { + // invalidate exampleplugin only archives in past + $invalidator = StaticContainer::get(ArchiveInvalidator::class); + $invalidator->markArchivesAsInvalidated([1], ['2007-04-05'], 'day', new Segment('', [1]), false, false, 'ExamplePlugin'); + + // track a visit in 2007-04-05 so it will archive (don't want to force archiving because then this test will take another 15 mins) + $tracker = Fixture::getTracker(1, '2007-04-05'); + $tracker->setUrl('http://example.com/test/url'); + Fixture::checkResponse($tracker->doTrackPageView('abcdefg')); + + // empty the list so nothing is invalidated during core:archive (so we only archive ExamplePlugin and not all plugins) + $invalidator->forgetRememberedArchivedReportsToInvalidate(1, Date::factory('2007-04-05')); + $output = $this->runArchivePhpCron(); $expectedInvalidations = []; @@ -118,6 +153,40 @@ class ArchiveCronTest extends SystemTestCase } } + /** + * @depends testArchivePhpCron + */ + public function testArchivePhpCronWithSingleReportRearchive() + { + // invalidate a report so we get a partial archive (using the metric that gets incremented each time it is archived) + // (do it after the last run so we don't end up just re-using the ExamplePlugin archive) + $invalidator = StaticContainer::get(ArchiveInvalidator::class); + $invalidator->markArchivesAsInvalidated([1], ['2007-04-05'], 'day', new Segment('', [1]), false, false, 'ExamplePlugin.ExamplePlugin_example_metric2'); + + $sequence = new Sequence('ExamplePlugin_archiveCount'); + $beforeCount = $sequence->getCurrentId(); + + $output = $this->runArchivePhpCron(['-vvv' => null]); + + $afterCount = $sequence->getCurrentId(); + + $this->assertNotEquals($beforeCount, $afterCount, 'example plugin archiving was not triggered'); + + $this->runApiTests('ExamplePlugin.getExampleArchivedMetric', [ + 'idSite' => 'all', + 'date' => '2007-04-05', + 'periods' => ['day', 'week'], + 'testSuffix' => '_singleMetric', + ]); + + // test that latest archives for ExamplePlugin are partial + $archiveValues = Db::fetchAll("SELECT value FROM " . ArchiveTableCreator::getNumericTable(Date::factory('2007-04-05')) + . " WHERE `name` = 'done.ExamplePlugin' ORDER BY ts_archived DESC LIMIT 8"); + $archiveValues = array_column($archiveValues, 'value'); + $archiveValues = array_unique($archiveValues); + $this->assertEquals([5], $archiveValues); + } + public function testArchivePhpCronArchivesFullRanges() { self::$fixture->getTestEnvironment()->overrideConfig('General', 'enable_browser_archiving_triggering', 0); @@ -133,9 +202,11 @@ class ArchiveCronTest extends SystemTestCase $expectedInvalidations = []; $invalidationEntries = $this->getInvalidatedArchiveTableEntries(); + $invalidationEntries = array_filter($invalidationEntries, function ($entry) { return $entry['period'] == 5; }); + $this->assertEquals($expectedInvalidations, $invalidationEntries); $this->runApiTests(array( @@ -143,8 +214,7 @@ class ArchiveCronTest extends SystemTestCase array('idSite' => '1', 'date' => '2012-08-09,2012-08-13', 'periods' => array('range'), - 'testSuffix' => '_range_archive' - ) + 'testSuffix' => '_range_archive') ); } diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest__ExamplePlugin.getExampleArchivedMetric_day.xml b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest__ExamplePlugin.getExampleArchivedMetric_day.xml new file mode 100644 index 0000000000..2e1a657b1f --- /dev/null +++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest__ExamplePlugin.getExampleArchivedMetric_day.xml @@ -0,0 +1,9 @@ +<?xml version="1.0" encoding="utf-8" ?> +<results> + <result idSite="1"> + <ExamplePlugin_example_metric>3382</ExamplePlugin_example_metric> + <ExamplePlugin_example_metric2>51</ExamplePlugin_example_metric2> + </result> + <result idSite="2" /> + <result idSite="3" /> +</results>
\ No newline at end of file diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_archive_php_cron_output.txt b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_archive_php_cron_output.txt index e8b09e0083..81032bb1a7 100644 --- a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_archive_php_cron_output.txt +++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_archive_php_cron_output.txt @@ -10,90 +10,86 @@ try 'php archive.php --url=http://your.piwik/path' ------------------------------------------------------- -INFO [2020-06-25 09:52:47] 10351 --------------------------- -INFO [2020-06-25 09:52:47] 10351 INIT -INFO [2020-06-25 09:52:47] 10351 Running Matomo 4.0.0-b2 as Super User -INFO [2020-06-25 09:52:47] 10351 --------------------------- -INFO [2020-06-25 09:52:47] 10351 NOTES -INFO [2020-06-25 09:52:47] 10351 - If you execute this script at least once per hour (or more often) in a crontab, you may disable 'Browser trigger archiving' in Matomo UI > Settings > General Settings. -INFO [2020-06-25 09:52:47] 10351 See the doc at: https://matomo.org/docs/setup-auto-archiving/ -INFO [2020-06-25 09:52:47] 10351 - Async process archiving supported, using CliMulti. -INFO [2020-06-25 09:52:47] 10351 - Reports for today will be processed at most every 900 seconds. You can change this value in Matomo UI > Settings > General Settings. -INFO [2020-06-25 09:52:47] 10351 --------------------------- -INFO [2020-06-25 09:52:47] 10351 START -INFO [2020-06-25 09:52:47] 10351 Starting Matomo reports archiving... -INFO [2020-06-25 09:52:47] 10351 Checking for queued invalidations... -INFO [2020-06-25 09:52:47] 10351 Will invalidate archived reports for 2014-03-13 for following websites ids: 3,1 -INFO [2020-06-25 09:52:47] 10351 Will invalidate archived reports for 2014-03-12 for following websites ids: 1,3 -INFO [2020-06-25 09:52:47] 10351 Will invalidate archived reports for 2012-09-30 for following websites ids: 1 -INFO [2020-06-25 09:52:47] 10351 Will invalidate archived reports for 2012-08-19 for following websites ids: 1 -INFO [2020-06-25 09:52:47] 10351 Will invalidate archived reports for 2012-08-15 for following websites ids: 1,3,2 -INFO [2020-06-25 09:52:47] 10351 Will invalidate archived reports for 2012-08-11 for following websites ids: 1 -INFO [2020-06-25 09:52:47] 10351 Will invalidate archived reports for 2012-08-10 for following websites ids: 1 -INFO [2020-06-25 09:52:47] 10351 Will invalidate archived reports for 2012-08-09 for following websites ids: 1 -INFO [2020-06-25 09:52:47] 10351 Segment "browserCode==IE" was created or changed recently and will therefore archive today (for site ID = 1) -INFO [2020-06-25 09:52:47] 10351 Segment "visitCount<=5;visitorType!=non-existing-type;daysSinceFirstVisit<=50" was created or changed recently and will therefore archive today (for site ID = 1) -INFO [2020-06-25 09:52:47] 10351 Segment "visitCount<=5;visitorType!=re%2C%3Btest%20is%20encoded;daysSinceFirstVisit<=50" was created or changed recently and will therefore archive today (for site ID = 1) -INFO [2020-06-25 09:52:47] 10351 Done invalidating -INFO [2020-06-25 09:52:47] 10351 Start processing archives for site 1. -INFO [2020-06-25 09:52:50] 10351 Archived website id 1, period = day, date = 2014-03-13, segment = '', 11 visits found. Time elapsed: 1.477s -INFO [2020-06-25 09:52:50] 10351 Archived website id 1, period = day, date = 2014-03-12, segment = '', 1 visits found. Time elapsed: 1.477s -INFO [2020-06-25 09:52:50] 10351 Archived website id 1, period = day, date = 2012-09-30, segment = '', 1 visits found. Time elapsed: 1.477s -INFO [2020-06-25 09:52:51] 10351 Archived website id 1, period = week, date = 2014-03-10, segment = '', 12 visits found. Time elapsed: 1.116s -INFO [2020-06-25 09:52:51] 10351 Archived website id 1, period = week, date = 2012-09-24, segment = '', 1 visits found. Time elapsed: 1.151s -INFO [2020-06-25 09:52:51] 10351 Archived website id 1, period = week, date = 2012-08-13, segment = '', 3 visits found. Time elapsed: 1.425s -INFO [2020-06-25 09:52:54] 10351 Archived website id 1, period = month, date = 2014-03-01, segment = '', 12 visits found. Time elapsed: 1.115s -INFO [2020-06-25 09:52:54] 10351 Archived website id 1, period = month, date = 2012-09-01, segment = '', 1 visits found. Time elapsed: 1.115s -INFO [2020-06-25 09:52:54] 10351 Archived website id 1, period = month, date = 2012-08-01, segment = '', 33 visits found. Time elapsed: 1.936s -INFO [2020-06-25 09:52:55] 10351 Archived website id 1, period = year, date = 2014-01-01, segment = '', 12 visits found. Time elapsed: 0.919s -INFO [2020-06-25 09:52:55] 10351 Archived website id 1, period = year, date = 2012-01-01, segment = '', 34 visits found. Time elapsed: 0.919s -INFO [2020-06-25 09:52:59] 10351 Archived website id 1, period = year, date = 2014-01-01, segment = 'browserCode%3D%3DIE', 1 visits found. Time elapsed: 3.088s -INFO [2020-06-25 09:52:59] 10351 Archived website id 1, period = year, date = 2014-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 12 visits found. Time elapsed: 3.967s -INFO [2020-06-25 09:52:59] 10351 Archived website id 1, period = year, date = 2014-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 12 visits found. Time elapsed: 3.967s -INFO [2020-06-25 09:53:01] 10351 Archived website id 1, period = month, date = 2012-09-01, segment = 'browserCode%3D%3DIE', 0 visits found. Time elapsed: 1.113s -INFO [2020-06-25 09:53:01] 10351 Archived website id 1, period = month, date = 2012-09-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 1 visits found. Time elapsed: 1.996s -INFO [2020-06-25 09:53:01] 10351 Archived website id 1, period = month, date = 2012-09-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 1 visits found. Time elapsed: 1.996s -INFO [2020-06-25 09:53:04] 10351 Archived website id 1, period = week, date = 2012-08-13, segment = 'browserCode%3D%3DIE', 1 visits found. Time elapsed: 1.807s -INFO [2020-06-25 09:53:04] 10351 Archived website id 1, period = week, date = 2012-08-13, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 3 visits found. Time elapsed: 1.859s -INFO [2020-06-25 09:53:04] 10351 Archived website id 1, period = week, date = 2012-08-13, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 3 visits found. Time elapsed: 2.129s -INFO [2020-06-25 09:53:05] 10351 Archived website id 1, period = day, date = 2012-08-11, segment = 'browserCode%3D%3DIE', 1 visits found. Time elapsed: 1.130s -INFO [2020-06-25 09:53:05] 10351 Archived website id 1, period = day, date = 2012-08-11, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 11 visits found. Time elapsed: 1.130s -INFO [2020-06-25 09:53:05] 10351 Archived website id 1, period = day, date = 2012-08-11, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 11 visits found. Time elapsed: 1.403s -INFO [2020-06-25 09:53:06] 10351 Archived website id 1, period = day, date = 2012-08-10, segment = 'browserCode%3D%3DIE', 3 visits found. Time elapsed: 1.114s -INFO [2020-06-25 09:53:06] 10351 Archived website id 1, period = day, date = 2012-08-10, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 8 visits found. Time elapsed: 1.426s -INFO [2020-06-25 09:53:06] 10351 Archived website id 1, period = day, date = 2012-08-10, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 8 visits found. Time elapsed: 1.426s -INFO [2020-06-25 09:53:08] 10351 Archived website id 1, period = day, date = 2012-08-09, segment = 'browserCode%3D%3DIE', 1 visits found. Time elapsed: 1.099s -INFO [2020-06-25 09:53:08] 10351 Archived website id 1, period = day, date = 2012-08-09, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 11 visits found. Time elapsed: 1.417s -INFO [2020-06-25 09:53:08] 10351 Archived website id 1, period = day, date = 2012-08-09, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 11 visits found. Time elapsed: 1.417s -INFO [2020-06-25 09:53:10] 10351 Archived website id 1, period = week, date = 2012-08-06, segment = 'browserCode%3D%3DIE', 5 visits found. Time elapsed: 1.442s -INFO [2020-06-25 09:53:10] 10351 Archived website id 1, period = week, date = 2012-08-06, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 30 visits found. Time elapsed: 1.442s -INFO [2020-06-25 09:53:10] 10351 Archived website id 1, period = week, date = 2012-08-06, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 30 visits found. Time elapsed: 1.714s -INFO [2020-06-25 09:53:11] 10351 Archived website id 1, period = month, date = 2012-08-01, segment = 'browserCode%3D%3DIE', 6 visits found. Time elapsed: 1.403s -INFO [2020-06-25 09:53:11] 10351 Archived website id 1, period = month, date = 2012-08-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 33 visits found. Time elapsed: 1.403s -INFO [2020-06-25 09:53:11] 10351 Archived website id 1, period = month, date = 2012-08-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 33 visits found. Time elapsed: 1.676s -INFO [2020-06-25 09:53:13] 10351 Archived website id 1, period = year, date = 2012-01-01, segment = 'browserCode%3D%3DIE', 6 visits found. Time elapsed: 1.421s -INFO [2020-06-25 09:53:13] 10351 Archived website id 1, period = year, date = 2012-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 34 visits found. Time elapsed: 1.728s -INFO [2020-06-25 09:53:13] 10351 Archived website id 1, period = year, date = 2012-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 34 visits found. Time elapsed: 1.728s -INFO [2020-06-25 09:53:13] 10351 Finished archiving for site 1, 38 API requests, Time elapsed: 25.964s [1 / 3 done] -INFO [2020-06-25 09:53:13] 10351 Start processing archives for site 2. -INFO [2020-06-25 09:53:14] 10351 Archived website id 2, period = day, date = 2012-08-15, segment = '', 1 visits found. Time elapsed: 0.556s -INFO [2020-06-25 09:53:14] 10351 Archived website id 2, period = week, date = 2012-08-13, segment = '', 1 visits found. Time elapsed: 0.555s -INFO [2020-06-25 09:53:15] 10351 Archived website id 2, period = month, date = 2012-08-01, segment = '', 1 visits found. Time elapsed: 0.555s -INFO [2020-06-25 09:53:16] 10351 Archived website id 2, period = year, date = 2012-01-01, segment = '', 1 visits found. Time elapsed: 0.554s -INFO [2020-06-25 09:53:16] 10351 Finished archiving for site 2, 4 API requests, Time elapsed: 2.374s [2 / 3 done] -INFO [2020-06-25 09:53:16] 10351 Start processing archives for site 3. -INFO [2020-06-25 09:53:17] 10351 Archived website id 3, period = day, date = 2014-03-13, segment = '', 11 visits found. Time elapsed: 1.116s -INFO [2020-06-25 09:53:17] 10351 Archived website id 3, period = day, date = 2014-03-12, segment = '', 1 visits found. Time elapsed: 1.116s -INFO [2020-06-25 09:53:17] 10351 Archived website id 3, period = day, date = 2012-08-15, segment = '', 3 visits found. Time elapsed: 1.116s -INFO [2020-06-25 09:53:18] 10351 Archived website id 3, period = week, date = 2014-03-10, segment = '', 12 visits found. Time elapsed: 0.930s -INFO [2020-06-25 09:53:18] 10351 Archived website id 3, period = week, date = 2012-08-13, segment = '', 3 visits found. Time elapsed: 0.930s -INFO [2020-06-25 09:53:19] 10351 Archived website id 3, period = month, date = 2014-03-01, segment = '', 12 visits found. Time elapsed: 0.932s -INFO [2020-06-25 09:53:19] 10351 Archived website id 3, period = month, date = 2012-08-01, segment = '', 3 visits found. Time elapsed: 0.932s -INFO [2020-06-25 09:53:20] 10351 Archived website id 3, period = year, date = 2014-01-01, segment = '', 12 visits found. Time elapsed: 0.923s -INFO [2020-06-25 09:53:20] 10351 Archived website id 3, period = year, date = 2012-01-01, segment = '', 3 visits found. Time elapsed: 0.923s -INFO [2020-06-25 09:53:20] 10351 Finished archiving for site 3, 9 API requests, Time elapsed: 4.088s [3 / 3 done] -INFO [2020-06-25 09:53:20] 10351 --------------------------- -INFO [2020-06-25 09:53:20] 10351 SCHEDULED TASKS -INFO [2020-06-25 09:53:20] 10351 Starting Scheduled tasks... -INFO [2020-06-25 09:53:20] 10351 done -INFO [2020-06-25 09:53:20] 10351 ---------------------------
\ No newline at end of file +INFO [2020-07-06 21:59:34] 10307 --------------------------- +INFO [2020-07-06 21:59:34] 10307 INIT +INFO [2020-07-06 21:59:34] 10307 Running Matomo 4.0.0-b2 as Super User +INFO [2020-07-06 21:59:34] 10307 --------------------------- +INFO [2020-07-06 21:59:34] 10307 NOTES +INFO [2020-07-06 21:59:34] 10307 - If you execute this script at least once per hour (or more often) in a crontab, you may disable 'Browser trigger archiving' in Matomo UI > Settings > General Settings. +INFO [2020-07-06 21:59:34] 10307 See the doc at: https://matomo.org/docs/setup-auto-archiving/ +INFO [2020-07-06 21:59:34] 10307 - Async process archiving supported, using CliMulti. +INFO [2020-07-06 21:59:34] 10307 - Reports for today will be processed at most every 900 seconds. You can change this value in Matomo UI > Settings > General Settings. +INFO [2020-07-06 21:59:34] 10307 --------------------------- +INFO [2020-07-06 21:59:34] 10307 START +INFO [2020-07-06 21:59:34] 10307 Starting Matomo reports archiving... +INFO [2020-07-06 21:59:34] 10307 Checking for queued invalidations... +INFO [2020-07-06 21:59:34] 10307 Will invalidate archived reports for 2014-03-13 for following websites ids: 1,3 +INFO [2020-07-06 21:59:34] 10307 Will invalidate archived reports for 2014-03-12 for following websites ids: 3,1 +INFO [2020-07-06 21:59:34] 10307 Will invalidate archived reports for 2012-09-30 for following websites ids: 1 +INFO [2020-07-06 21:59:34] 10307 Will invalidate archived reports for 2012-08-19 for following websites ids: 1 +INFO [2020-07-06 21:59:34] 10307 Will invalidate archived reports for 2012-08-15 for following websites ids: 2,1,3 +INFO [2020-07-06 21:59:34] 10307 Will invalidate archived reports for 2012-08-11 for following websites ids: 1 +INFO [2020-07-06 21:59:34] 10307 Will invalidate archived reports for 2012-08-10 for following websites ids: 1 +INFO [2020-07-06 21:59:34] 10307 Will invalidate archived reports for 2012-08-09 for following websites ids: 1 +INFO [2020-07-06 21:59:34] 10307 Segment "browserCode==IE" was created or changed recently and will therefore archive today (for site ID = 1) +INFO [2020-07-06 21:59:34] 10307 Segment "visitCount<=5;visitorType!=non-existing-type;daysSinceFirstVisit<=50" was created or changed recently and will therefore archive today (for site ID = 1) +INFO [2020-07-06 21:59:34] 10307 Segment "visitCount<=5;visitorType!=re%2C%3Btest%20is%20encoded;daysSinceFirstVisit<=50" was created or changed recently and will therefore archive today (for site ID = 1) +INFO [2020-07-06 21:59:34] 10307 Done invalidating +INFO [2020-07-06 21:59:34] 10307 Start processing archives for site 1. +INFO [2020-07-06 21:59:37] 10307 Archived website id 1, period = day, date = 2014-03-13, segment = '', 11 visits found. Time elapsed: 1.124s +INFO [2020-07-06 21:59:37] 10307 Archived website id 1, period = day, date = 2014-03-12, segment = '', 1 visits found. Time elapsed: 1.124s +INFO [2020-07-06 21:59:37] 10307 Archived website id 1, period = day, date = 2012-09-30, segment = '', 1 visits found. Time elapsed: 1.124s +INFO [2020-07-06 21:59:38] 10307 Archived website id 1, period = week, date = 2014-03-10, segment = '', 12 visits found. Time elapsed: 1.761s +INFO [2020-07-06 21:59:38] 10307 Archived website id 1, period = week, date = 2012-09-24, segment = '', 1 visits found. Time elapsed: 1.761s +INFO [2020-07-06 21:59:38] 10307 Archived website id 1, period = day, date = 2012-08-19, segment = '', 1 visits found. Time elapsed: 1.761s +INFO [2020-07-06 21:59:40] 10307 Archived website id 1, period = month, date = 2014-03-01, segment = '', 12 visits found. Time elapsed: 1.743s +INFO [2020-07-06 21:59:40] 10307 Archived website id 1, period = month, date = 2012-09-01, segment = '', 1 visits found. Time elapsed: 1.743s +INFO [2020-07-06 21:59:40] 10307 Archived website id 1, period = day, date = 2012-08-15, segment = '', 2 visits found. Time elapsed: 1.743s +INFO [2020-07-06 21:59:43] 10307 Archived website id 1, period = year, date = 2014-01-01, segment = '', 12 visits found. Time elapsed: 2.510s +INFO [2020-07-06 21:59:43] 10307 Archived website id 1, period = week, date = 2012-08-13, segment = '', 3 visits found. Time elapsed: 2.510s +INFO [2020-07-06 21:59:43] 10307 Archived website id 1, period = day, date = 2012-08-11, segment = '', 11 visits found. Time elapsed: 2.510s +INFO [2020-07-06 21:59:47] 10307 Archived website id 1, period = year, date = 2014-01-01, segment = 'browserCode%3D%3DIE', 1 visits found. Time elapsed: 3.015s +INFO [2020-07-06 21:59:47] 10307 Archived website id 1, period = year, date = 2012-01-01, segment = 'browserCode%3D%3DIE', 6 visits found. Time elapsed: 4.118s +INFO [2020-07-06 21:59:47] 10307 Archived website id 1, period = day, date = 2007-04-05, segment = '', plugin = ExamplePlugin, 1 visits found. Time elapsed: 4.118s +INFO [2020-07-06 21:59:53] 10307 Archived website id 1, period = year, date = 2014-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 12 visits found. Time elapsed: 4.601s +INFO [2020-07-06 21:59:53] 10307 Archived website id 1, period = year, date = 2012-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dnon-existing-type%3BdaysSinceFirstVisit%3C%3D50', 34 visits found. Time elapsed: 5.707s +INFO [2020-07-06 21:59:53] 10307 Archived website id 1, period = week, date = 2007-04-02, segment = '', plugin = ExamplePlugin, 1 visits found. Time elapsed: 5.707s +INFO [2020-07-06 21:59:59] 10307 Archived website id 1, period = year, date = 2014-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 12 visits found. Time elapsed: 3.018s +INFO [2020-07-06 21:59:59] 10307 Archived website id 1, period = year, date = 2012-01-01, segment = 'visitCount%3C%3D5%3BvisitorType%21%3Dre%252C%253Btest%2520is%2520encoded%3BdaysSinceFirstVisit%3C%3D50', 34 visits found. Time elapsed: 5.246s +INFO [2020-07-06 21:59:59] 10307 Archived website id 1, period = month, date = 2007-04-01, segment = '', plugin = ExamplePlugin, 1 visits found. Time elapsed: 5.246s +INFO [2020-07-06 22:00:00] 10307 Archived website id 1, period = day, date = 2012-08-10, segment = '', 8 visits found. Time elapsed: 0.632s +INFO [2020-07-06 22:00:00] 10307 Archived website id 1, period = day, date = 2012-08-09, segment = '', 11 visits found. Time elapsed: 0.916s +INFO [2020-07-06 22:00:01] 10307 Archived website id 1, period = week, date = 2012-08-06, segment = '', 30 visits found. Time elapsed: 0.827s +INFO [2020-07-06 22:00:02] 10307 Archived website id 1, period = month, date = 2012-08-01, segment = '', 33 visits found. Time elapsed: 1.104s +INFO [2020-07-06 22:00:03] 10307 Archived website id 1, period = year, date = 2012-01-01, segment = '', 34 visits found. Time elapsed: 1.102s +INFO [2020-07-06 22:00:03] 10307 Archived website id 1, period = year, date = 2007-01-01, segment = '', plugin = ExamplePlugin, 1 visits found. Time elapsed: 0.564s +INFO [2020-07-06 22:00:03] 10307 Finished archiving for site 1, 27 API requests, Time elapsed: 28.981s [1 / 3 done] +INFO [2020-07-06 22:00:04] 10307 Start processing archives for site 2. +INFO [2020-07-06 22:00:04] 10307 Archived website id 2, period = day, date = 2012-08-15, segment = '', 1 visits found. Time elapsed: 0.549s +INFO [2020-07-06 22:00:05] 10307 Archived website id 2, period = week, date = 2012-08-13, segment = '', 1 visits found. Time elapsed: 0.826s +INFO [2020-07-06 22:00:06] 10307 Archived website id 2, period = month, date = 2012-08-01, segment = '', 1 visits found. Time elapsed: 0.825s +INFO [2020-07-06 22:00:07] 10307 Archived website id 2, period = year, date = 2012-01-01, segment = '', 1 visits found. Time elapsed: 1.110s +INFO [2020-07-06 22:00:07] 10307 Finished archiving for site 2, 4 API requests, Time elapsed: 3.469s [2 / 3 done] +INFO [2020-07-06 22:00:07] 10307 Start processing archives for site 3. +INFO [2020-07-06 22:00:08] 10307 Archived website id 3, period = day, date = 2014-03-13, segment = '', 11 visits found. Time elapsed: 0.834s +INFO [2020-07-06 22:00:08] 10307 Archived website id 3, period = day, date = 2014-03-12, segment = '', 1 visits found. Time elapsed: 0.834s +INFO [2020-07-06 22:00:08] 10307 Archived website id 3, period = day, date = 2012-08-15, segment = '', 3 visits found. Time elapsed: 1.115s +INFO [2020-07-06 22:00:09] 10307 Archived website id 3, period = week, date = 2014-03-10, segment = '', 12 visits found. Time elapsed: 1.257s +INFO [2020-07-06 22:00:09] 10307 Archived website id 3, period = week, date = 2012-08-13, segment = '', 3 visits found. Time elapsed: 1.257s +INFO [2020-07-06 22:00:11] 10307 Archived website id 3, period = month, date = 2014-03-01, segment = '', 12 visits found. Time elapsed: 1.266s +INFO [2020-07-06 22:00:11] 10307 Archived website id 3, period = month, date = 2012-08-01, segment = '', 3 visits found. Time elapsed: 1.818s +INFO [2020-07-06 22:00:13] 10307 Archived website id 3, period = year, date = 2014-01-01, segment = '', 12 visits found. Time elapsed: 1.899s +INFO [2020-07-06 22:00:13] 10307 Archived website id 3, period = year, date = 2012-01-01, segment = '', 3 visits found. Time elapsed: 1.920s +INFO [2020-07-06 22:00:13] 10307 Finished archiving for site 3, 9 API requests, Time elapsed: 6.303s [3 / 3 done] +INFO [2020-07-06 22:00:13] 10307 Done archiving! +INFO [2020-07-06 22:00:13] 10307 --------------------------- +INFO [2020-07-06 22:00:13] 10307 SUMMARY +INFO [2020-07-06 22:00:13] 10307 Processed 40 archives. +INFO [2020-07-06 22:00:13] 10307 Total API requests: 40 +INFO [2020-07-06 22:00:13] 10307 done: 40 req, 39477 ms, no error +INFO [2020-07-06 22:00:13] 10307 Time elapsed: 39.477s +INFO [2020-07-06 22:00:13] 10307 --------------------------- +INFO [2020-07-06 22:00:13] 10307 SCHEDULED TASKS +INFO [2020-07-06 22:00:13] 10307 Starting Scheduled tasks... +INFO [2020-07-06 22:00:13] 10307 done +INFO [2020-07-06 22:00:13] 10307 ---------------------------
\ No newline at end of file diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_examplePluginNoMetricsBecauseNoOtherPluginsArchived_noOptions__Actions.get_day.xml b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_examplePluginNoMetricsBecauseNoOtherPluginsArchived_noOptions__Actions.get_day.xml new file mode 100644 index 0000000000..dd52dc4ff4 --- /dev/null +++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_examplePluginNoMetricsBecauseNoOtherPluginsArchived_noOptions__Actions.get_day.xml @@ -0,0 +1,6 @@ +<?xml version="1.0" encoding="utf-8" ?> +<results> + <result idSite="1" /> + <result idSite="2" /> + <result idSite="3" /> +</results>
\ No newline at end of file diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_examplePluginNoMetricsBecauseNoOtherPluginsArchived_noOptions__Actions.get_week.xml b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_examplePluginNoMetricsBecauseNoOtherPluginsArchived_noOptions__Actions.get_week.xml new file mode 100644 index 0000000000..dd52dc4ff4 --- /dev/null +++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_examplePluginNoMetricsBecauseNoOtherPluginsArchived_noOptions__Actions.get_week.xml @@ -0,0 +1,6 @@ +<?xml version="1.0" encoding="utf-8" ?> +<results> + <result idSite="1" /> + <result idSite="2" /> + <result idSite="3" /> +</results>
\ No newline at end of file diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_noOptions__ExamplePlugin.getExampleArchivedMetric_day.xml b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_noOptions__ExamplePlugin.getExampleArchivedMetric_day.xml new file mode 100644 index 0000000000..bbba10abb2 --- /dev/null +++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_noOptions__ExamplePlugin.getExampleArchivedMetric_day.xml @@ -0,0 +1,9 @@ +<?xml version="1.0" encoding="utf-8" ?> +<results> + <result idSite="1"> + <ExamplePlugin_example_metric>3382</ExamplePlugin_example_metric> + <ExamplePlugin_example_metric2>60</ExamplePlugin_example_metric2> + </result> + <result idSite="2" /> + <result idSite="3" /> +</results>
\ No newline at end of file diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_noOptions__ExamplePlugin.getExampleArchivedMetric_week.xml b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_noOptions__ExamplePlugin.getExampleArchivedMetric_week.xml new file mode 100644 index 0000000000..bbba10abb2 --- /dev/null +++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_noOptions__ExamplePlugin.getExampleArchivedMetric_week.xml @@ -0,0 +1,9 @@ +<?xml version="1.0" encoding="utf-8" ?> +<results> + <result idSite="1"> + <ExamplePlugin_example_metric>3382</ExamplePlugin_example_metric> + <ExamplePlugin_example_metric2>60</ExamplePlugin_example_metric2> + </result> + <result idSite="2" /> + <result idSite="3" /> +</results>
\ No newline at end of file diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_singleMetric__ExamplePlugin.getExampleArchivedMetric_day.xml b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_singleMetric__ExamplePlugin.getExampleArchivedMetric_day.xml new file mode 100644 index 0000000000..4214cca45d --- /dev/null +++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_singleMetric__ExamplePlugin.getExampleArchivedMetric_day.xml @@ -0,0 +1,9 @@ +<?xml version="1.0" encoding="utf-8" ?> +<results> + <result idSite="1"> + <ExamplePlugin_example_metric>3382</ExamplePlugin_example_metric> + <ExamplePlugin_example_metric2>90</ExamplePlugin_example_metric2> + </result> + <result idSite="2" /> + <result idSite="3" /> +</results>
\ No newline at end of file diff --git a/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_singleMetric__ExamplePlugin.getExampleArchivedMetric_week.xml b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_singleMetric__ExamplePlugin.getExampleArchivedMetric_week.xml new file mode 100644 index 0000000000..4214cca45d --- /dev/null +++ b/plugins/CoreConsole/tests/System/expected/test_ArchiveCronTest_singleMetric__ExamplePlugin.getExampleArchivedMetric_week.xml @@ -0,0 +1,9 @@ +<?xml version="1.0" encoding="utf-8" ?> +<results> + <result idSite="1"> + <ExamplePlugin_example_metric>3382</ExamplePlugin_example_metric> + <ExamplePlugin_example_metric2>90</ExamplePlugin_example_metric2> + </result> + <result idSite="2" /> + <result idSite="3" /> +</results>
\ No newline at end of file |