diff options
| author | diosmosis <diosmosis@users.noreply.github.com> | 2020-08-04 05:59:58 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2020-08-04 05:59:58 +0300 |
| commit | f5e9420a987340b036fa342e876ab92e314f4ec7 (patch) | |
| tree | 2267e9eafe8b6577c4f7d3219d39c284a90677c6 /tests | |
| parent | 2394c8c954d46d1ca9fe8d1304405e7fd6727c89 (diff) | |
allow invalidating plugin archives only and archiving past data for plugins (#15889)
* Adding initial new code for cron archive rewrite.
* first pass at removing unused CronArchive code.
* unfinished commit
* fill out archiveinvalidator code
* getting some tests to pass
* unfinished commit
* fixing part of test
* Another test fix.
* another sql change
* fix broken merge or something else that went wrong
* Couple more fixes and extra logs.
* Fixing enough issues to get core archive command to run completely.
* Fix and log change.
* Fixed more segment/test related issues for CronArchiveTest. Includes optimization for no visits for period + segment process from handling.
* another optimization and possible build fix
* no visit optimization
* test fix
* Implement archiving_custom_ranges logic w/ queue based implementation
* fixes to get archivecrontest to work
* add logic to invalidate today period
* fix optimization and some tests
* Fixing more tests.
* Fixing more tests
* debug travis failure
* more test fixes
* more test fixes, removing more unneeded code, handling some TODOs
* Handle more TODOs including creating ArchiveFilter class for some cli options.
* tests and todos
* idarchives are specific to table + start on archivefilter tests
* one test
* more TODOs and tests
* more tests and todo taken care of
* handle more todos
* fixing more tests
* fix comment
* make sure autoarchiving is enabled for segments when cron archive picks them up
* Fixing test.
* apply more pr feedback
* order by date1 asc
* quick refactor
* use batch insert instead of createDummyArchives
* apply rest of pr feedback
* add removed events, add new test, fix an issue (when deleting idarchives older than do not lump all segments together).
* re-add fixed/shared siteids
* fix tests
* incomplete commit
* Insert archive entries into archive_invalidations table.
* Use invalidations table in core:archive and get ArchiveCronTest to pass.
* fixing some tests
* debugging travis
* fix more tests & remove DONE_IN_PROGRESS which is no longer used.
* fix more tests
* Allow forcing plugin specific archive in core:archive.
* When querying from archive data use all available archives including "all" archives and plugin specific archives.
* Adding some code for invalidating specific plugin archives.
* Get archive invalidation test to pass.
* add plugin capability to invalidate command
* Handle plugin only archives in core:archive.
* Add Archive test and get ArchiveCronTest to pass.
* update some expected files
* Fix some more tests.
* incomplete commit
* allow invalidating individual reports
* adding more API for DONE_PARTIAL support
* get archivecrontest to pass
* add archive processor tests
* fix some test randomnes
* when purging keep latest partial archives if there is no newer whole archive
* add rearchivereport method + some unfinished tests
* Add archiveReports API method, fix race condition in test, when archiving single report, always ignore inserting other reports.
* require archivers to handle partial archives themselves entirely instead of trying to do it automatically and allow requested report to be any string
* couple fixes
* Use core config option for last N montsh to invalidate.
* Add test for ArchiveSelector method.
* Ignore archives w/ deactivated plugins.
* Refactor queue looping into new QueueConsumer class.
* apply more review feedback + another fix
* invalidate segments too in reArchiveReport w/ etsts
* remove DONE_IN_PROGRESS, no longer used. use new status in query and add queue consumer test.
* forgot to add file
* delete old unneeded archives when finalizing a new one.
* tweak invalidation archive description
* add plugin archiving tests and get them to pass
* fix test
* many fixes
* fix another test
* update expected test files
* fix more tests
* last test fixes hopefully
* tweak log
* In case a column already exists, do not try to add it in an AddColumns migration or the entire migration will fail and no columns will be added.
* try to fix tests again
* fix again?
* apply some review feedback + fix test
* fix test
* fix another test
* couple fixes
* Remove extra param.
* apply pr feedback
* check for usable archive before invalidating and before initiating archiving
* fixing tests
* fixing tests
* fixing tests
* fix another test issue
* fix archiveinvalidator test
* fix one test and debug another
* more debugging
* fix test
* use twig
* remove no longer needed change
* add back previous logic
* fix tracking is not working
* apply pr feedback and add tests
* fixing tests
* update submodule
* debugging random travis failure
* update test
* more debugging
* more debugging
* another attempt at debugging
* Lets try this fix
* trying to fix the build
* debug
* simpler test
* fix test
* fix test
* fix test
* fix test
* fix test failure
* update screenshots
* update screenshots
Co-authored-by: Thomas Steur <tsteur@users.noreply.github.com>
Diffstat (limited to 'tests')
24 files changed, 2871 insertions, 285 deletions
diff --git a/tests/PHPUnit/Fixtures/RawArchiveDataWithTempAndInvalidated.php b/tests/PHPUnit/Fixtures/RawArchiveDataWithTempAndInvalidated.php index 36c1a0ba49..408a090d18 100644 --- a/tests/PHPUnit/Fixtures/RawArchiveDataWithTempAndInvalidated.php +++ b/tests/PHPUnit/Fixtures/RawArchiveDataWithTempAndInvalidated.php @@ -70,7 +70,6 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture 'ts_archived' => '2015-02-15 08:12:13' ), - // valid temporary array( // only valid 'idarchive' => 5, @@ -262,12 +261,35 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture 'ts_archived' => '2015-02-09 14:13:14' ), + // old done partial (should be purged) + array( + 'idarchive' => 22, + 'idsite' => 1, + 'name' => 'done', + 'value' => ArchiveWriter::DONE_PARTIAL, + 'date1' => '2015-02-10', + 'date2' => '2015-02-10', + 'period' => 1, + 'ts_archived' => '2015-02-11 10:13:14' + ), + + // new done partial (should not be purged) + array( + 'idarchive' => 23, + 'idsite' => 1, + 'name' => 'done', + 'value' => ArchiveWriter::DONE_PARTIAL, + 'date1' => '2015-02-10', + 'date2' => '2015-02-10', + 'period' => 1, + 'ts_archived' => '2015-02-11 16:13:14' + ), ); private static $segmentArchiveData = array( array( - 'idarchive' => 22, + 'idarchive' => 24, 'idsite' => 1, 'name' => 'doneeb5d2797aedd15d819b1a20425982850', // Raw segment = abcd1234abcd5678 'value' => ArchiveWriter::DONE_OK, @@ -277,7 +299,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture 'ts_archived' => '2015-02-03 12:12:12' ), array( - 'idarchive' => 23, + 'idarchive' => 25, 'idsite' => 1, 'name' => 'doneeb5d2797aedd15d819b1a20425982850.MyPlugin', // Raw segment = abcd1234abcd5678 'value' => ArchiveWriter::DONE_OK, @@ -287,7 +309,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture 'ts_archived' => '2015-02-03 12:12:12' ), array( - 'idarchive' => 24, + 'idarchive' => 26, 'idsite' => 2, 'name' => 'doneeb5d2797aedd15d819b1a20425982850', // Raw segment = abcd1234abcd5678 'value' => ArchiveWriter::DONE_OK, @@ -297,7 +319,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture 'ts_archived' => '2015-02-03 12:12:12' ), array( - 'idarchive' => 25, + 'idarchive' => 27, 'idsite' => 2, 'name' => 'doneeb5d2797aedd15d819b1a20425982850.MyPlugin', // Raw segment = abcd1234abcd5678 'value' => ArchiveWriter::DONE_OK, @@ -307,7 +329,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture 'ts_archived' => '2015-02-03 12:12:12' ), array( - 'idarchive' => 26, + 'idarchive' => 28, 'idsite' => 1, 'name' => 'done1e39a89fcc269acc36bd4d7c742763ed', // Raw segment = 9876fedc5432abcd 'value' => ArchiveWriter::DONE_OK, @@ -317,7 +339,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture 'ts_archived' => '2015-02-03 12:12:12' ), array( - 'idarchive' => 27, + 'idarchive' => 29, 'idsite' => 2, 'name' => 'done00c6ee2e21a7548de6260cf72c4f4b5b', // Raw segment = hash1 'value' => ArchiveWriter::DONE_OK, @@ -327,7 +349,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture 'ts_archived' => '2015-02-03 12:12:12' ), array( - 'idarchive' => 28, + 'idarchive' => 30, 'idsite' => 2, 'name' => 'done58833651db311ba4bc11cb26b1900b0f', // Raw segment = hash2 'value' => ArchiveWriter::DONE_OK, @@ -337,7 +359,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture 'ts_archived' => '2015-02-03 12:12:12' ), array( - 'idarchive' => 29, + 'idarchive' => 31, 'idsite' => 2, 'name' => 'done58833651db311ba4bc11cb26b1900b0f.MyPlugin', // Raw segment = hash2 'value' => ArchiveWriter::DONE_OK, @@ -347,7 +369,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture 'ts_archived' => '2015-02-03 12:12:12' ), array( - 'idarchive' => 30, + 'idarchive' => 32, 'idsite' => 2, 'name' => 'done1a4ead8b39d17dfe89418452c9bba770', // Raw segment = hash3 'value' => ArchiveWriter::DONE_OK, @@ -357,7 +379,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture 'ts_archived' => '2015-02-03 12:12:12' ), array( - 'idarchive' => 31, + 'idarchive' => 33, 'idsite' => 2, 'name' => 'done1a4ead8b39d17dfe89418452c9bba770', // Raw segment = hash3 'value' => ArchiveWriter::DONE_OK, @@ -371,7 +393,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture public static $dummyArchiveDataNoInvalidated = [ // two archives w/ DONE_OK for a new site (no invalidated archives for site) [ - 'idarchive' => 32, + 'idarchive' => 34, 'idsite' => 4, 'name' => 'done', 'value' => ArchiveWriter::DONE_OK, @@ -381,7 +403,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture 'ts_archived' => '2015-02-27 10:12:12' ], [ - 'idarchive' => 33, + 'idarchive' => 35, 'idsite' => 4, 'name' => 'done', 'value' => ArchiveWriter::DONE_OK, @@ -558,4 +580,13 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture $expectedExistingArchives = array(15); $this->assertArchivesExist($expectedExistingArchives, $date); } + + public function assertPartialArchivesPurged(Date $date) + { + $expectedPurgedArchives = [22]; + $this->assertArchivesDoNotExist($expectedPurgedArchives, $date); + + $expectedExistingArchives = [23]; + $this->assertArchivesExist($expectedExistingArchives, $date); + } }
\ No newline at end of file diff --git a/tests/PHPUnit/Framework/Mock/FakeLogger.php b/tests/PHPUnit/Framework/Mock/FakeLogger.php index 3ee8a182d0..7cc89eca55 100644 --- a/tests/PHPUnit/Framework/Mock/FakeLogger.php +++ b/tests/PHPUnit/Framework/Mock/FakeLogger.php @@ -25,6 +25,10 @@ class FakeLogger extends AbstractLogger implements LoggerInterface public function log($level, $message, array $context = array()) { + if (strpos($message, 'Running command') !== false) { + return; + } + $record = $this->processor->__invoke(array('message' => $message, 'context' => $context)); $this->output .= $record['message'] . PHP_EOL; diff --git a/tests/PHPUnit/Integration/Archive/ArchivePurgerTest.php b/tests/PHPUnit/Integration/Archive/ArchivePurgerTest.php index 4c2d6c6dc0..c73feeb7b1 100644 --- a/tests/PHPUnit/Integration/Archive/ArchivePurgerTest.php +++ b/tests/PHPUnit/Integration/Archive/ArchivePurgerTest.php @@ -99,8 +99,9 @@ class ArchivePurgerTest extends IntegrationTestCase self::$fixture->assertInvalidatedArchivesPurged($this->february); self::$fixture->assertInvalidatedArchivesNotPurged($this->january); + self::$fixture->assertPartialArchivesPurged($this->february); - $this->assertEquals(9 * RawArchiveDataWithTempAndInvalidated::ROWS_PER_ARCHIVE, $deletedRowCount); + $this->assertEquals(10 * RawArchiveDataWithTempAndInvalidated::ROWS_PER_ARCHIVE, $deletedRowCount); $this->checkNoDuplicateArchives(); } @@ -142,7 +143,7 @@ class ArchivePurgerTest extends IntegrationTestCase //Archive #29 also has a deleted segment but it's before the purge threshold so it stays for now. $deletedRowCount = $this->archivePurger->purgeDeletedSegmentArchives($this->january, $segmentsToDelete); $this->assertEquals(4 * RawArchiveDataWithTempAndInvalidated::ROWS_PER_ARCHIVE, $deletedRowCount); - self::$fixture->assertArchivesDoNotExist(array(24, 25, 26, 30), $this->january); + self::$fixture->assertArchivesDoNotExist(array(26, 27, 28, 32), $this->january); } public function test_purgeNoSegmentArchives_preservesSingleSiteSegmentArchivesForDeletedAllSiteSegment() @@ -158,7 +159,7 @@ class ArchivePurgerTest extends IntegrationTestCase // Archives for idsite=1 should be purged, but those for idsite=2 can stay $deletedRowCount = $this->archivePurger->purgeDeletedSegmentArchives($this->january, $segmentsToDelete); $this->assertEquals(2 * RawArchiveDataWithTempAndInvalidated::ROWS_PER_ARCHIVE, $deletedRowCount); - self::$fixture->assertArchivesDoNotExist(array(22, 23), $this->january); + self::$fixture->assertArchivesDoNotExist(array(24, 25), $this->january); } public function test_purgeNoSegmentArchives_blankSegmentName() diff --git a/tests/PHPUnit/Integration/ArchiveProcessingTest.php b/tests/PHPUnit/Integration/ArchiveProcessingTest.php index 9e63b2536d..b1df5a98a9 100644 --- a/tests/PHPUnit/Integration/ArchiveProcessingTest.php +++ b/tests/PHPUnit/Integration/ArchiveProcessingTest.php @@ -10,10 +10,14 @@ namespace Piwik\Tests\Integration; use Exception; use Piwik\Access; +use Piwik\Archive; use Piwik\ArchiveProcessor; use Piwik\ArchiveProcessor\Rules; use Piwik\Common; use Piwik\DataAccess\ArchiveTableCreator; +use Piwik\DataAccess\ArchiveWriter; +use Piwik\DataAccess\LogAggregator; +use Piwik\DataTable; use Piwik\Date; use Piwik\Db; use Piwik\Db\BatchInsert; @@ -24,6 +28,7 @@ use Piwik\Plugins\SitesManager\API; use Piwik\Segment; use Piwik\SettingsServer; use Piwik\Site; +use Piwik\Tests\Framework\Fixture; use Piwik\Tests\Framework\Mock\FakeAccess; use Piwik\Tests\Framework\TestCase\IntegrationTestCase; @@ -63,15 +68,7 @@ class ArchiveProcessingTest extends IntegrationTestCase */ private function _createWebsite($timezone = 'UTC') { - $idSite = API::getInstance()->addSite( - "site1", - array("http://piwik.net"), - $ecommerce = 0, - $siteSearch = 1, $searchKeywordParameters = null, $searchCategoryParameters = null, - $excludedIps = "", - $excludedQueryParameters = "", - $timezone); - + $idSite = Fixture::createWebsite('2013-03-04', 0, false, false, 1, null, null, $timezone); Site::clearCache(); return new Site($idSite); } @@ -89,12 +86,60 @@ class ArchiveProcessingTest extends IntegrationTestCase $site = $this->_createWebsite($siteTimezone); $date = Date::factory($dateLabel); $period = Period\Factory::build($periodLabel, $date); - $segment = new Segment('', $site->getId(), $period->getDateStart(), $period->getDateEnd()); + $segment = new Segment('', [$site->getId()], $period->getDateStart(), $period->getDateEnd()); $params = new ArchiveProcessor\Parameters($site, $period, $segment); return new ArchiveProcessorTest($params); } + private function _createArchiveProcessorInst($periodLabel, $dateLabel, $idSite, $archiveOnly = false, $plugin = false) + { + $period = Period\Factory::build($periodLabel, $dateLabel); + $segment = new Segment('', [$idSite]); + + $params = new ArchiveProcessor\Parameters(new Site($idSite), $period, $segment); + if ($archiveOnly) { + $params->setRequestedPlugin($plugin); + $params->setArchiveOnlyReport($archiveOnly); + } + $archiveWriter = new ArchiveWriter($params); + $logAggregator = new LogAggregator($params); + $archiveProcessor = new class($params, $archiveWriter, $logAggregator) extends ArchiveProcessor { + private $captureInserts = false; + private $capturedInserts = []; + + public function captureInserts() + { + $this->captureInserts = true; + } + + public function insertNumericRecord($name, $value) + { + if ($this->captureInserts) { + $this->capturedInserts[] = [$name, $value]; + } else { + parent::insertNumericRecord($name, $value); + } + } + + public function insertBlobRecord($name, $values) + { + if ($this->captureInserts) { + $this->capturedInserts[] = [$name, $values]; + } else { + parent::insertBlobRecord($name, $values); + } + } + + public function getCapturedInserts() + { + return $this->capturedInserts; + } + }; + + return [$archiveProcessor, $archiveWriter, $params]; + } + /** * test of validity of an archive, for a month not finished */ @@ -103,7 +148,7 @@ class ArchiveProcessingTest extends IntegrationTestCase $siteTimezone = 'UTC+10'; $now = time(); // this test fails in the last 10 hours of the last day of the month - if(date('m', $now) != date('m', $now + 10 * 3600)) { + if (date('m', $now) != date('m', $now + 10 * 3600)) { $this->markTestSkipped('testInitCurrentMonth will fail in the last hours of the month, skipping...'); } @@ -120,7 +165,7 @@ class ArchiveProcessingTest extends IntegrationTestCase { // $messageIfFails = Date::factory($expected)->getDatetime() . " != " . Date::factory($processed)->getDatetime(); $messageIfFails = "Expected [$expected] but got [$processed]"; - $this->assertTrue( abs($expected-$processed) <=4 , $messageIfFails); + $this->assertTrue(abs($expected - $processed) <= 4, $messageIfFails); } /** @@ -277,12 +322,12 @@ class ArchiveProcessingTest extends IntegrationTestCase $skippedOnce = true; $this->fail( 'Performance notice: LOAD DATA [LOCAL] INFILE query is not working, so Piwik will fallback to using plain INSERTs ' - . ' which will result in a slightly slower Archiving process.' - . ". \n" - . ' The error Messages from MySQL were: ' - . $didWeUseBulk - . "\n\n Learn more how to enable LOAD LOCAL DATA INFILE see the Mysql doc (http://dev.mysql.com/doc/refman/5.0/en/load-data-local.html) " - . "\n or ask in this Piwik ticket (https://github.com/matomo-org/matomo/issues/3605)" + . ' which will result in a slightly slower Archiving process.' + . ". \n" + . ' The error Messages from MySQL were: ' + . $didWeUseBulk + . "\n\n Learn more how to enable LOAD LOCAL DATA INFILE see the Mysql doc (http://dev.mysql.com/doc/refman/5.0/en/load-data-local.html) " + . "\n or ask in this Piwik ticket (https://github.com/matomo-org/matomo/issues/3605)" ); } return $didWeUseBulk; @@ -364,6 +409,232 @@ class ArchiveProcessingTest extends IntegrationTestCase $this->fail('Exception expected'); } + public function test_aggregateNumericMetrics_aggregatesCorrectly() + { + $allMetrics = [ + '2015-02-03' => [ + 'nb_visits' => 2, + 'max_actions' => 3, + ], + '2015-02-04' => [ + 'nb_visits' => 2, + 'max_actions' => 4, + ], + '2015-02-05' => [ + 'nb_visits' => 2, + 'max_actions' => 1, + ], + ]; + + $site = $this->_createWebsite('UTC'); + + foreach ($allMetrics as $date => $metrics) { + /** @var ArchiveWriter $archiveWriter */ + list($archiveProcessor, $archiveWriter, $params) = $this->_createArchiveProcessorInst('day', $date, $site->getId()); + $archiveWriter->initNewArchive(); + + $archiveProcessor->insertNumericRecords($metrics); + + $archiveWriter->finalizeArchive(); + } + + /** @var ArchiveProcessor $archiveProcessor */ + list($archiveProcessor, $archiveWriter, $params) = $this->_createArchiveProcessorInst('week', '2015-02-03', $site->getId()); + $archiveWriter->initNewArchive(); + + $archiveProcessor->captureInserts(); + $archiveProcessor->aggregateNumericMetrics(['nb_visits', 'max_actions']); + + $archiveWriter->finalizeArchive(); + + $capturedInserts = $archiveProcessor->getCapturedInserts(); + + $expected = [ + [ + 'nb_visits', + 6, + ], + [ + 'max_actions', + 4, + ] + ]; + + $this->assertEquals($expected, $capturedInserts); + } + + public function test_aggregateNumericMetrics_handlesPartialArchives() + { + $allMetrics = [ + '2015-02-03' => [ + 'nb_visits' => 2, + 'max_actions' => 1, + ], + '2015-02-04' => [ + 'nb_visits' => 2, + 'max_actions' => 3, + ], + '2015-02-05' => [ + 'nb_visits' => 2, + 'max_actions' => 4, + ], + ]; + + $site = $this->_createWebsite('UTC'); + + foreach ($allMetrics as $date => $metrics) { + /** @var ArchiveWriter $archiveWriter */ + list($archiveProcessor, $archiveWriter) = $this->_createArchiveProcessorInst('day', $date, $site->getId()); + $archiveWriter->initNewArchive(); + + $archiveProcessor->insertNumericRecords($metrics); + + $archiveWriter->finalizeArchive(); + } + + /** @var ArchiveProcessor $archiveProcessor */ + list($archiveProcessor, $archiveWriter, $params) = $this->_createArchiveProcessorInst('week', '2015-02-03', $site->getId(), 'nb_visits', 'VisitsSummary'); + $params->setIsPartialArchive(true); + $idArchive = $archiveWriter->initNewArchive(); + + $archiveProcessor->captureInserts(); + $archiveProcessor->aggregateNumericMetrics(['nb_visits']); + + $archiveWriter->finalizeArchive(); + + $capturedInserts = $archiveProcessor->getCapturedInserts(); + + $expected = [ + [ + 'nb_visits', + 6, + ], + ]; + + $archiveDoneFlag = Db::fetchOne("SELECT `value` FROM " . ArchiveTableCreator::getNumericTable(Date::factory('2015-02-03')) . " WHERE idarchive = ? AND name LIKE 'done%'", [$idArchive]); + $this->assertEquals(ArchiveWriter::DONE_PARTIAL, $archiveDoneFlag); + + $this->assertEquals($expected, $capturedInserts); + } + + public function test_aggregateDataTableRecords_aggregatesCorrectly() + { + $table1 = new DataTable(); + $table1->addRowsFromSimpleArray([ + ['label' => 'a', 'nb_visits' => 5, 'nb_actions' => 1], + ['label' => 'b', 'nb_visits' => 3, 'nb_actions' => 1], + ]); + $table2 = new DataTable(); + $table2->addRowsFromSimpleArray([ + ['label' => 'a', 'nb_visits' => 2, 'nb_actions' => 2], + ]); + $table3 = new DataTable(); + $table3->addRowsFromSimpleArray([ + ['label' => 'b', 'nb_visits' => 4, 'nb_actions' => 3], + ]); + + $tables = [ + '2015-02-03' => $table1, + '2015-02-04' => $table2, + '2015-02-05' => $table3, + ]; + + $site = $this->_createWebsite('UTC'); + + foreach ($tables as $date => $table) { + /** @var ArchiveWriter $archiveWriter */ + list($archiveProcessor, $archiveWriter) = $this->_createArchiveProcessorInst('day', $date, $site->getId()); + $archiveWriter->initNewArchive(); + + $tableSerialized = $table->getSerialized(); + $archiveProcessor->insertBlobRecord('Actions_test_value', $tableSerialized); + + $archiveWriter->finalizeArchive(); + } + + list($archiveProcessor, $archiveWriter) = $this->_createArchiveProcessorInst('week', '2015-02-03', $site->getId()); + $archiveWriter->initNewArchive(); + + $archiveProcessor->captureInserts(); + $archiveProcessor->aggregateDataTableRecords('Actions_test_value'); + + $archiveWriter->finalizeArchive(); + + $capturedInserts = $archiveProcessor->getCapturedInserts(); + $capturedInsertTable = DataTable::fromSerializedArray($capturedInserts[0][1][0]); + $capturedInsertTable = $this->getXml($capturedInsertTable); + + $expectedXml = <<<END +<?xml version="1.0" encoding="utf-8" ?> +<result> + <row> + <label>a</label> + <nb_visits>7</nb_visits> + <nb_actions>3</nb_actions> + </row> + <row> + <label>b</label> + <nb_visits>7</nb_visits> + <nb_actions>4</nb_actions> + </row> +</result> +END; + + $this->assertEquals($expectedXml, $capturedInsertTable); + } + + public function test_aggregateDataTableRecords_handlesPartialArchives() + { + $table1 = new DataTable(); + $table1->addRowsFromSimpleArray([ + ['label' => 'a', 'nb_visits' => 5, 'nb_actions' => 1], + ['label' => 'b', 'nb_visits' => 3, 'nb_actions' => 1], + ]); + $table2 = new DataTable(); + $table2->addRowsFromSimpleArray([ + ['label' => 'a', 'nb_visits' => 2, 'nb_actions' => 2], + ]); + $table3 = new DataTable(); + $table3->addRowsFromSimpleArray([ + ['label' => 'b', 'nb_visits' => 4, 'nb_actions' => 3], + ]); + + $tables = [ + '2015-02-03' => $table1, + '2015-02-04' => $table2, + '2015-02-05' => $table3, + ]; + + $site = $this->_createWebsite('UTC'); + + foreach ($tables as $date => $table) { + /** @var ArchiveWriter $archiveWriter */ + list($archiveProcessor, $archiveWriter) = $this->_createArchiveProcessorInst('day', $date, $site->getId()); + $archiveWriter->initNewArchive(); + + $tableSerialized = $table->getSerialized(); + $archiveProcessor->insertBlobRecord('Actions_test_value', $tableSerialized); + + $archiveWriter->finalizeArchive(); + } + + /** @var ArchiveProcessor $archiveProcessor */ + list($archiveProcessor, $archiveWriter, $params) = $this->_createArchiveProcessorInst('week', '2015-02-03', $site->getId(), 'Actions_test_value', 'VisitsSummary'); + $params->setIsPartialArchive(true); + $idArchive = $archiveWriter->initNewArchive(); + + $archiveProcessor->captureInserts(); + $archiveProcessor->aggregateDataTableRecords('Actions_test_value'); + + $archiveWriter->finalizeArchive(); + + $capturedInserts = $archiveProcessor->getCapturedInserts(); + $this->assertNotEmpty($capturedInserts); + + $archiveDoneFlag = Db::fetchOne("SELECT `value` FROM " . ArchiveTableCreator::getNumericTable(Date::factory('2015-02-03')) . " WHERE idarchive = ? AND name LIKE 'done%'", [$idArchive]); + $this->assertEquals(ArchiveWriter::DONE_PARTIAL, $archiveDoneFlag); + } + protected function _checkTableIsExpected($table, $data) { $fetched = Db::fetchAll('SELECT * FROM ' . $table); @@ -450,4 +721,11 @@ class ArchiveProcessingTest extends IntegrationTestCase 'Piwik\Access' => new FakeAccess() ); } + + private function getXml(DataTable $capturedInsertTable) + { + $xml = new DataTable\Renderer\Xml(); + $xml->setTable($capturedInsertTable); + return $xml->render(); + } } diff --git a/tests/PHPUnit/Integration/ArchiveProcessor/LoaderTest.php b/tests/PHPUnit/Integration/ArchiveProcessor/LoaderTest.php index 2b5f18c238..3690ef45ea 100644 --- a/tests/PHPUnit/Integration/ArchiveProcessor/LoaderTest.php +++ b/tests/PHPUnit/Integration/ArchiveProcessor/LoaderTest.php @@ -13,6 +13,7 @@ namespace Piwik\Tests\Integration\ArchiveProcessor; use Piwik\Archive\ArchiveInvalidator; use Piwik\ArchiveProcessor\Parameters; use Piwik\ArchiveProcessor\Loader; +use Piwik\ArchiveProcessor\PluginsArchiver; use Piwik\Common; use Piwik\Config; use Piwik\Container\StaticContainer; @@ -22,7 +23,9 @@ use Piwik\Date; use Piwik\Db; use Piwik\Period\Factory; use Piwik\Piwik; +use Piwik\Plugins\ExamplePlugin\Archiver; use Piwik\Segment; +use Piwik\Sequence; use Piwik\Site; use Piwik\Tests\Framework\Fixture; use Piwik\Tests\Framework\TestCase\IntegrationTestCase; @@ -37,6 +40,760 @@ class LoaderTest extends IntegrationTestCase Fixture::createWebsite('2012-02-03 00:00:00'); } + /** + * @dataProvider getTestDataForArchiving + */ + public function test_pluginOnlyArchivingCreatesAndReusesCorrectArchives($archiveData, $params, $expectedArchives, $archiveTwice) + { + $_GET['pluginOnly'] = 1; + $_GET['trigger'] = 'archivephp'; + + Date::$now = strtotime('2018-03-04 05:00:00'); + + list($idSite, $period, $date, $segment, $plugin, $report) = $params; + + $t = Fixture::getTracker($idSite, $date); + $t->setUrl('http://slkdfj.com'); + $t->doTrackPageView('alsdkjf'); + + $params = new Parameters(new Site($idSite), Factory::build($period, $date), new Segment($segment, [$idSite])); + $params->setRequestedPlugin($plugin); + if ($report) { + $params->setArchiveOnlyReport($report); + } + + $this->insertArchiveData($archiveData); + + $loader = new Loader($params); + $loader->prepareArchive($params->getRequestedPlugin()); + + if ($archiveTwice) { + if (is_array($archiveTwice)) { + list($idSite2, $period2, $date2, $segment2, $plugin2, $report2) = $archiveTwice; + + $params2 = new Parameters(new Site($idSite2), Factory::build($period2, $date2), new Segment($segment2, [$idSite2])); + $params2->setRequestedPlugin($plugin2); + if ($report2) { + $params2->setArchiveOnlyReport($report2); + } + } else { + $params2 = $params; + } + + $loader2 = new Loader($params2); + $loader2->prepareArchive($params->getRequestedPlugin()); + } + + $actualArchives = $this->getArchives(); + if ($actualArchives != $expectedArchives) { + var_export($actualArchives); + } + $this->assertEquals($expectedArchives, $actualArchives); + } + + public function getTestDataForArchiving() + { + $pluginSpecificArchive = [1, 'day', '2018-03-03', '', 'ExamplePlugin', false]; + + $reportSpecificArchive1 = [1, 'day', '2018-03-03', '', 'ExamplePlugin', Archiver::EXAMPLEPLUGIN_METRIC_NAME]; + $reportSpecificArchive2 = [1, 'day', '2018-03-03', '', 'ExamplePlugin', Archiver::EXAMPLEPLUGIN_CONST_METRIC_NAME]; + + $unloadedPluginArchive = [1, 'day', '2018-03-03', '', 'MyImaginaryPlugin', false]; + + return [ + // no archive, archive specific plugin + [ + [], + $pluginSpecificArchive, + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'bounce_count', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.VisitsSummary', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'max_actions', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_actions', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_uniq_visitors', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_visits', + 'value' => '1', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.ExamplePlugin', + 'value' => '1', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'ExamplePlugin_example_metric', + 'value' => '-603', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'ExamplePlugin_example_metric2', + 'value' => '55', + ), + ), + false, + ], + + // all plugins, recent, archive specific plugin + [ + [ + ['idarchive' => 1, 'idsite' => 1, 'date1' => '2018-03-03', 'date2' => '2018-03-03', 'period' => 1, 'name' => 'done', 'value' => ArchiveWriter::DONE_OK, 'ts_archived' => '2018-03-04 04:50:00'], + ['idarchive' => 1, 'idsite' => 1, 'date1' => '2018-03-03', 'date2' => '2018-03-03', 'period' => 1, 'name' => 'nb_visits', 'value' => 12, 'ts_archived' => '2018-03-04 04:50:00'], + ['idarchive' => 1, 'idsite' => 1, 'date1' => '2018-03-03', 'date2' => '2018-03-03', 'period' => 1, 'name' => 'nb_visits_converted', 'value' => 3, 'ts_archived' => '2018-03-04 04:50:00'], + ], + $pluginSpecificArchive, + array ( // done archive already exists and is recent, so we don't archive the plugin + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_visits', + 'value' => '12', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_visits_converted', + 'value' => '3', + ), + ), + false, + ], + + // visitssummary, recent, archive specific plugin + [ + [ + ['idarchive' => 1, 'idsite' => 1, 'date1' => '2018-03-03', 'date2' => '2018-03-03', 'period' => 1, 'name' => 'done.VisitsSummary', 'value' => ArchiveWriter::DONE_OK, 'ts_archived' => '2018-03-04 04:50:00'], + ['idarchive' => 1, 'idsite' => 1, 'date1' => '2018-03-03', 'date2' => '2018-03-03', 'period' => 1, 'name' => 'nb_visits', 'value' => 12, 'ts_archived' => '2018-03-04 04:50:00'], + ['idarchive' => 1, 'idsite' => 1, 'date1' => '2018-03-03', 'date2' => '2018-03-03', 'period' => 1, 'name' => 'nb_visits_converted', 'value' => 3, 'ts_archived' => '2018-03-04 04:50:00'], + ], + $pluginSpecificArchive, + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.VisitsSummary', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_visits', + 'value' => '12', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_visits_converted', + 'value' => '3', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.ExamplePlugin', + 'value' => '1', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'ExamplePlugin_example_metric', + 'value' => '-603', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'ExamplePlugin_example_metric2', + 'value' => '55', + ), + ), + false, + ], + + // all plugins, old, archive specific plugin + [ + [ + ['idarchive' => 1, 'idsite' => 1, 'date1' => '2018-03-03', 'date2' => '2018-03-03', 'period' => 1, 'name' => 'done', 'value' => ArchiveWriter::DONE_OK, 'ts_archived' => '2018-03-01 04:50:00'], + ['idarchive' => 1, 'idsite' => 1, 'date1' => '2018-03-03', 'date2' => '2018-03-03', 'period' => 1, 'name' => 'nb_visits', 'value' => 12, 'ts_archived' => '2018-03-01 04:50:00'], + ['idarchive' => 1, 'idsite' => 1, 'date1' => '2018-03-03', 'date2' => '2018-03-03', 'period' => 1, 'name' => 'nb_visits_converted', 'value' => 3, 'ts_archived' => '2018-03-01 04:50:00'], + ], + $pluginSpecificArchive, + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_visits', + 'value' => '12', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_visits_converted', + 'value' => '3', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.ExamplePlugin', + 'value' => '1', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'ExamplePlugin_example_metric', + 'value' => '-603', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'ExamplePlugin_example_metric2', + 'value' => '55', + ), + ), + false, + ], + + // visitssummary, old, archive specific plugin + [ + [ + ['idarchive' => 1, 'idsite' => 1, 'date1' => '2018-03-03', 'date2' => '2018-03-03', 'period' => 1, 'name' => 'done.VisitsSummary', 'value' => ArchiveWriter::DONE_OK, 'ts_archived' => '2018-03-01 04:50:00'], + ['idarchive' => 1, 'idsite' => 1, 'date1' => '2018-03-03', 'date2' => '2018-03-03', 'period' => 1, 'name' => 'nb_visits', 'value' => 12, 'ts_archived' => '2018-03-01 04:50:00'], + ['idarchive' => 1, 'idsite' => 1, 'date1' => '2018-03-03', 'date2' => '2018-03-03', 'period' => 1, 'name' => 'nb_visits_converted', 'value' => 3, 'ts_archived' => '2018-03-01 04:50:00'], + ], + $pluginSpecificArchive, + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.VisitsSummary', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_visits', + 'value' => '12', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_visits_converted', + 'value' => '3', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.ExamplePlugin', + 'value' => '1', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'ExamplePlugin_example_metric', + 'value' => '-603', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'ExamplePlugin_example_metric2', + 'value' => '55', + ), + ), + false, + ], + + // no archive, archive specific plugin, archive specific plugin again + [ + [], + $pluginSpecificArchive, + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'bounce_count', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.VisitsSummary', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'max_actions', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_actions', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_uniq_visitors', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_visits', + 'value' => '1', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.ExamplePlugin', + 'value' => '1', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'ExamplePlugin_example_metric', + 'value' => '-603', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'ExamplePlugin_example_metric2', + 'value' => '55', + ), + ), + true, + ], + + // no archive, archive specific report, archive specific report again + [ + [], + $reportSpecificArchive1, + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'bounce_count', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.VisitsSummary', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'max_actions', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_actions', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_uniq_visitors', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_visits', + 'value' => '1', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.ExamplePlugin', + 'value' => '5', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'ExamplePlugin_example_metric', + 'value' => '-603', + ), + array ( + 'idarchive' => '3', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.ExamplePlugin', + 'value' => '5', + ), + array ( + 'idarchive' => '3', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'ExamplePlugin_example_metric', + 'value' => '-603', + ), + ), + true, + ], + + // no archive, archive specific report, archive different report again + [ + [], + $reportSpecificArchive1, + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'bounce_count', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.VisitsSummary', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'max_actions', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_actions', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_uniq_visitors', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_visits', + 'value' => '1', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.ExamplePlugin', + 'value' => '5', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'ExamplePlugin_example_metric', + 'value' => '-603', + ), + array ( + 'idarchive' => '3', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.ExamplePlugin', + 'value' => '5', + ), + array ( + 'idarchive' => '3', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'ExamplePlugin_example_metric2', + 'value' => '55', + ), + ), + $reportSpecificArchive2, + ], + + // no archive, unloaded plugin + [ + [], + $unloadedPluginArchive, + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'bounce_count', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.VisitsSummary', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'max_actions', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_actions', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_uniq_visitors', + 'value' => '1', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'nb_visits', + 'value' => '1', + ), + array ( + 'idarchive' => '2', + 'idsite' => '1', + 'date1' => '2018-03-03', + 'date2' => '2018-03-03', + 'period' => '1', + 'name' => 'done.MyImaginaryPlugin', + 'value' => '1', + ), + ), + false, + ], + ]; + } + public function test_loadExistingArchiveIdFromDb_returnsFalsesIfNoArchiveFound() { $params = new Parameters(new Site(1), Factory::build('day', '2015-03-03'), new Segment('', [1])); @@ -95,7 +852,7 @@ class LoaderTest extends IntegrationTestCase $this->assertNotEmpty($archiveInfo[4]); unset($archiveInfo[4]); - $this->assertEquals(['1', '10', '0', true], $archiveInfo); + $this->assertEquals([['1'], '10', '0', true], $archiveInfo); } public function test_loadExistingArchiveIdFromDb_returnsArchiveIfForACurrentPeriod_AndNewEnough() @@ -110,7 +867,7 @@ class LoaderTest extends IntegrationTestCase $this->assertNotEmpty($archiveInfo[4]); unset($archiveInfo[4]); - $this->assertEquals(['1', '10', '0', true], $archiveInfo); + $this->assertEquals([['1'], '10', '0', true], $archiveInfo); } public function test_loadExistingArchiveIdFromDb_returnsNoArchiveIfForACurrentPeriod_AndNoneAreNewEnough() @@ -331,6 +1088,26 @@ class LoaderTest extends IntegrationTestCase $this->assertFalse($loader->canSkipThisArchive()); } + public function test_forcePluginArchiving_createsPluginSpecificArchive() + { + $_GET['trigger'] = 'archivephp'; + $_GET['pluginOnly'] = '1'; + + $params = new Parameters(new Site(1), Factory::build('day', '2016-02-03'), new Segment('', [1])); + $loader = new Loader($params); + + $tracker = Fixture::getTracker(1, '2016-02-03 00:00:00'); + $tracker->setUrl('http://example.org/abc'); + Fixture::checkResponse($tracker->doTrackPageView('abc')); + + $idArchive = $loader->prepareArchive('Actions')[0]; + $this->assertNotEmpty($idArchive); + + $table = ArchiveTableCreator::getNumericTable(Date::factory('2016-02-03')); + $doneFlag = Db::fetchOne("SELECT `name` FROM `$table` WHERE `name` LIKE 'done%' AND idarchive IN (" . implode($idArchive, ',') . ")"); + $this->assertEquals('done.Actions', $doneFlag); + } + private function insertArchive(Parameters $params, $tsArchived = null, $visits = 10) { $archiveWriter = new ArchiveWriter($params); @@ -349,4 +1126,38 @@ class LoaderTest extends IntegrationTestCase parent::configureFixture($fixture); $fixture->createSuperUser = true; } -}
\ No newline at end of file + + private function insertArchiveData($archiveRows) + { + foreach ($archiveRows as $row) { + if (!empty($row['is_blob_data'])) { + $row['value'] = gzcompress($row['value']); + } + + $d = Date::factory($row['date1']); + $table = !empty($row['is_blob_data']) ? ArchiveTableCreator::getBlobTable($d) : ArchiveTableCreator::getNumericTable($d); + $tsArchived = isset($row['ts_archived']) ? $row['ts_archived'] : Date::now()->getDatetime(); + + Db::query("INSERT INTO `$table` (idarchive, idsite, period, date1, date2, `name`, `value`, ts_archived) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", + [$row['idarchive'], $row['idsite'], $row['period'], $row['date1'], $row['date2'], $row['name'], $row['value'], $tsArchived]); + } + + if (!empty($archiveRows)) { + $idarchives = array_column($archiveRows, 'idarchive'); + $max = max($idarchives); + + $seq = new Sequence(ArchiveTableCreator::getNumericTable(Date::factory($archiveRows[0]['date1']))); + $seq->create($max); + } + } + + private function getArchives() + { + $results = []; + foreach (ArchiveTableCreator::getTablesArchivesInstalled('numeric', true) as $table) { + $queryResults = Db::fetchAll("SELECT idarchive, idsite, date1, date2, period, `name`, `value` FROM `$table`"); + $results = array_merge($results, $queryResults); + } + return $results; + } +} diff --git a/tests/PHPUnit/Integration/ArchiveProcessor/PluginsArchiverTest.php b/tests/PHPUnit/Integration/ArchiveProcessor/PluginsArchiverTest.php index 08a3c1ab2a..8b45eac55a 100644 --- a/tests/PHPUnit/Integration/ArchiveProcessor/PluginsArchiverTest.php +++ b/tests/PHPUnit/Integration/ArchiveProcessor/PluginsArchiverTest.php @@ -8,13 +8,10 @@ namespace Piwik\Tests\Integration\Archive; use Piwik\ArchiveProcessor\PluginsArchiver; -use Piwik\Config; use Piwik\Piwik; use Piwik\Segment; use Piwik\Site; -use Piwik\Db; use Piwik\ArchiveProcessor\Parameters; -use Exception; use Piwik\Plugin\Archiver; use Piwik\Tests\Framework\Fixture; use Piwik\Tests\Framework\TestCase\IntegrationTestCase; @@ -111,5 +108,4 @@ class PluginsArchiverTest extends IntegrationTestCase $this->assertTrue(true); // pass } - } diff --git a/tests/PHPUnit/Integration/ArchiveTest.php b/tests/PHPUnit/Integration/ArchiveTest.php new file mode 100644 index 0000000000..b2a75fc528 --- /dev/null +++ b/tests/PHPUnit/Integration/ArchiveTest.php @@ -0,0 +1,91 @@ +<?php +/** + * Matomo - free/libre analytics platform + * + * @link https://matomo.org + * @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later + * + */ + +namespace PHPUnit\Integration; + +use Piwik\Archive; +use Piwik\ArchiveProcessor\Parameters; +use Piwik\ArchiveProcessor\Rules; +use Piwik\Config; +use Piwik\DataAccess\ArchiveWriter; +use Piwik\Period\Factory; +use Piwik\Segment; +use Piwik\Site; +use Piwik\Tests\Framework\Fixture; +use Piwik\Tests\Framework\TestCase\IntegrationTestCase; + +class ArchiveTest extends IntegrationTestCase +{ + public function setUp(): void + { + parent::setUp(); + + Fixture::createWebsite('2014-05-06'); + } + + public function test_pluginSpecificArchiveUsed_EvenIfAllArchiveExists_IfThereAreNoDataInAllArchive() + { + $idSite = 1; + + // insert all plugin archive + $params = new Parameters(new Site($idSite), Factory::build('day', '2014-05-07'), new Segment('', [$idSite])); + $archiveWriter = new ArchiveWriter($params); + $archiveWriter->initNewArchive(); + $archiveWriter->insertRecord('ExamplePlugin_archive1metric', 1); + $archiveWriter->insertRecord('ExamplePlugin_archive2metric', 5); + $archiveWriter->finalizeArchive(); + + // insert single plugin archive + $_GET['pluginOnly'] = 1; + $_GET['trigger'] = 'archivephp'; + + $params = new Parameters(new Site($idSite), Factory::build('day', '2014-05-07'), new Segment('', [$idSite])); + $params->setRequestedPlugin('ExamplePlugin'); + $params->onlyArchiveRequestedPlugin(); + $archiveWriter = new ArchiveWriter($params); + $archiveWriter->initNewArchive(); + $archiveWriter->insertRecord('ExamplePlugin_archive2metric', 2); + $archiveWriter->insertRecord('ExamplePlugin_archive3metric', 3); + $archiveWriter->finalizeArchive(); + + // insert single plugin archive + $params = new Parameters(new Site($idSite), Factory::build('day', '2014-05-07'), new Segment('', [$idSite])); + $params->setRequestedPlugin('ExamplePlugin'); + $params->onlyArchiveRequestedPlugin(); + $archiveWriter = new ArchiveWriter($params); + $archiveWriter->initNewArchive(); + $archiveWriter->insertRecord('ExamplePlugin_archive3metric', 7); + $archiveWriter->finalizeArchive(); + + $archive = Archive::build($idSite, 'day', '2014-05-07'); + $metrics = $archive->getNumeric(['ExamplePlugin_archive1metric', 'ExamplePlugin_archive2metric', 'ExamplePlugin_archive3metric']); + + $expected = [ + 'ExamplePlugin_archive1metric' => 0, + 'ExamplePlugin_archive2metric' => 0, + 'ExamplePlugin_archive3metric' => 7, + ]; + + $this->assertEquals($expected, $metrics); + } + + public function test_pluginSpecificArchiveUsed_EvenIfAllArchiveExists_IfThereAreNoDataInAllArchive_WithBrowserArchivingDisabled() + { + self::$fixture->getTestEnvironment()->overrideConfig('General', 'enable_browser_archiving_triggering', 0); + self::$fixture->getTestEnvironment()->overrideConfig('General', 'archiving_range_force_on_browser_request', 0); + self::$fixture->getTestEnvironment()->save(); + + Config::getInstance()->General['enable_browser_archiving_triggering'] = 0; + Config::getInstance()->General['archiving_range_force_on_browser_request'] = 0; + + $this->assertTrue(Rules::isArchivingDisabledFor([1], new Segment('', [1]), 'day')); + + $this->test_pluginSpecificArchiveUsed_EvenIfAllArchiveExists_IfThereAreNoDataInAllArchive(); + } +}
\ No newline at end of file diff --git a/tests/PHPUnit/Integration/ArchiveWithNoVisitsTest.php b/tests/PHPUnit/Integration/ArchiveWithNoVisitsTest.php index 799036f1b3..44d1f3ea80 100644 --- a/tests/PHPUnit/Integration/ArchiveWithNoVisitsTest.php +++ b/tests/PHPUnit/Integration/ArchiveWithNoVisitsTest.php @@ -59,7 +59,6 @@ class ArchiveWithNoVisitsTest extends IntegrationTestCase $this->assertEmpty(ArchiveWithNoVisitsTest_MockArchiver::$methodsCalled); } - // TODO: changed this and another test, will it be an issue? now, CronArchive.getIdSitesNotUsingTracker has to add the site for it to archive no matter what. public function test_getIdSitesToArchiveWhenNoVisits_DoesNotTriggerArchiving_IfSiteHasNoVisits() { // add our mock archiver instance @@ -78,7 +77,16 @@ class ArchiveWithNoVisitsTest extends IntegrationTestCase // initiate archiving and make sure both aggregate methods are called correctly VisitsSummaryAPI::getInstance()->get($idSite = 1, 'week', '2012-01-10'); - $expectedMethodCalls = array(); + $expectedMethodCalls = array( + 'aggregateDayReport', + 'aggregateDayReport', + 'aggregateDayReport', + 'aggregateDayReport', + 'aggregateDayReport', + 'aggregateDayReport', + 'aggregateDayReport', + 'aggregateMultipleReports', + ); $this->assertEquals($expectedMethodCalls, ArchiveWithNoVisitsTest_MockArchiver::$methodsCalled); } diff --git a/tests/PHPUnit/Integration/CronArchive/QueueConsumerTest.php b/tests/PHPUnit/Integration/CronArchive/QueueConsumerTest.php new file mode 100644 index 0000000000..1d14e457b9 --- /dev/null +++ b/tests/PHPUnit/Integration/CronArchive/QueueConsumerTest.php @@ -0,0 +1,549 @@ +<?php +/** + * Matomo - free/libre analytics platform + * + * @link https://matomo.org + * @license http://www.gnu.org/licenses/gpl-3.0.html GPL v3 or later + * + */ + +namespace Piwik\Tests\Integration\CronArchive; + +use Piwik\CliMulti\RequestParser; +use Piwik\Common; +use Piwik\Container\StaticContainer; +use Piwik\CronArchive; +use Piwik\CronArchive\FixedSiteIds; +use Piwik\CronArchive\QueueConsumer; +use Piwik\CronArchive\SegmentArchiving; +use Piwik\DataAccess\ArchiveTableCreator; +use Piwik\DataAccess\ArchiveWriter; +use Piwik\DataAccess\Model; +use Piwik\Date; +use Piwik\Db; +use Piwik\Piwik; +use Piwik\Plugins\SegmentEditor\API; +use Piwik\Segment; +use Piwik\Tests\Framework\Fixture; +use Piwik\Tests\Framework\TestCase\IntegrationTestCase; +use Psr\Log\LoggerInterface; + +class QueueConsumerTest extends IntegrationTestCase +{ + public function test_invalidateConsumeOrder() + { + Fixture::createWebsite('2015-02-03'); + Fixture::createWebsite('2020-04-06'); + Fixture::createWebsite('2010-04-06'); + + API::getInstance()->add('testegment', 'browserCode==IE', false, true); + + // force archiving so we don't skip those without visits + Piwik::addAction('Archiving.getIdSitesToArchiveWhenNoVisits', function (&$idSites) { + $idSites[] = 1; + $idSites[] = 2; + }); + + $cronArchive = new CronArchive(); + + $archiveFilter = $this->makeTestArchiveFilter(); + + $queueConsumer = new QueueConsumer( + StaticContainer::get(LoggerInterface::class), + new FixedSiteIds([1,2,3]), + 3, + 24, + new Model(), + new SegmentArchiving('beginning_of_time'), + $cronArchive, + new RequestParser(true), + $archiveFilter + ); + + $segmentHash = (new Segment('browserCode==IE', [1]))->getHash(); + + $invalidations = [ + ['idarchive' => 1, 'name' => 'done', 'idsite' => 1, 'date1' => '2018-03-04', 'date2' => '2018-03-04', 'period' => 1, 'report' => null], + ['idarchive' => 1, 'name' => 'done', 'idsite' => 1, 'date1' => '2018-03-07', 'date2' => '2018-03-07', 'period' => 1, 'report' => null], + ['idarchive' => 1, 'name' => 'done', 'idsite' => 1, 'date1' => '2018-03-08', 'date2' => '2018-03-08', 'period' => 1, 'report' => null], + ['idarchive' => 1, 'name' => 'done', 'idsite' => 1, 'date1' => '2018-03-06', 'date2' => '2018-03-06', 'period' => 1, 'report' => null], + ['idarchive' => 1, 'name' => 'done', 'idsite' => 1, 'date1' => '2018-03-01', 'date2' => '2018-03-31', 'period' => 3, 'report' => null], + ['idarchive' => 1, 'name' => 'done', 'idsite' => 1, 'date1' => '2018-03-04', 'date2' => '2018-03-11', 'period' => 2, 'report' => null], + + ['idarchive' => 1, 'name' => 'done.Actions', 'idsite' => 1, 'date1' => '2018-03-06', 'date2' => '2018-03-06', 'period' => 1, 'report' => 'testReport'], + ['idarchive' => 1, 'name' => 'done.Actions', 'idsite' => 1, 'date1' => '2018-03-01', 'date2' => '2018-03-31', 'period' => 3, 'report' => 'testReport'], + ['idarchive' => 1, 'name' => 'done.Actions', 'idsite' => 1, 'date1' => '2018-03-04', 'date2' => '2018-03-11', 'period' => 2, 'report' => 'testReport'], + + // some or all subperiods before site was created + ['idarchive' => 1, 'name' => 'done', 'idsite' => 2, 'date1' => '2020-04-04', 'date2' => '2020-04-04', 'period' => 1, 'report' => 'testReport'], + ['idarchive' => 1, 'name' => 'done', 'idsite' => 2, 'date1' => '2020-03-30', 'date2' => '2020-04-05', 'period' => 2, 'report' => 'testReport'], + + // segments + ['idarchive' => 1, 'name' => 'done' . $segmentHash, 'idsite' => 1, 'date1' => '2018-03-04', 'date2' => '2018-03-04', 'period' => 1, 'report' => null], + ['idarchive' => 1, 'name' => 'done' . $segmentHash, 'idsite' => 1, 'date1' => '2018-03-07', 'date2' => '2018-03-07', 'period' => 1, 'report' => null], + ['idarchive' => 1, 'name' => 'done' . $segmentHash, 'idsite' => 1, 'date1' => '2018-03-08', 'date2' => '2018-03-08', 'period' => 1, 'report' => null], + ['idarchive' => 1, 'name' => 'done' . $segmentHash, 'idsite' => 1, 'date1' => '2018-03-06', 'date2' => '2018-03-06', 'period' => 1, 'report' => null], + ['idarchive' => 1, 'name' => 'done' . $segmentHash, 'idsite' => 1, 'date1' => '2018-03-01', 'date2' => '2018-03-31', 'period' => 3, 'report' => null], + ['idarchive' => 1, 'name' => 'done' . $segmentHash, 'idsite' => 1, 'date1' => '2018-03-04', 'date2' => '2018-03-11', 'period' => 2, 'report' => null], + + // invalid plugin + ['idarchive' => 1, 'name' => 'done.MyPlugin', 'idsite' => 1, 'date1' => '2018-03-04', 'date2' => '2018-03-11', 'period' => 2, 'report' => 'testReport'], + + // duplicates + ['idarchive' => 1, 'name' => 'done', 'idsite' => 1, 'date1' => '2018-03-06', 'date2' => '2018-03-06', 'period' => 1, 'report' => null], + ['idarchive' => 1, 'name' => 'done', 'idsite' => 1, 'date1' => '2018-03-01', 'date2' => '2018-03-31', 'period' => 3, 'report' => null], + ]; + + shuffle($invalidations); + + $this->insertInvalidations($invalidations); + + $iteratedInvalidations = []; + while (true) { + $next = $queueConsumer->getNextArchivesToProcess(); + if ($next === null) { + break; + } + + foreach ($next as &$item) { + Db::query("UPDATE " . Common::prefixTable('archive_invalidations') . " SET status = 1 WHERE idinvalidation = ?", [$item['idinvalidation']]); + + unset($item['periodObj']); + unset($item['idinvalidation']); + } + + $iteratedInvalidations[] = $next; + } + + $expectedInvalidationsFound = [ + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-08', + 'date2' => '2018-03-08', + 'period' => '1', + 'name' => 'done', + 'report' => NULL, + 'plugin' => NULL, + 'segment' => '', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-07', + 'date2' => '2018-03-07', + 'period' => '1', + 'name' => 'done', + 'report' => NULL, + 'plugin' => NULL, + 'segment' => '', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-06', + 'date2' => '2018-03-06', + 'period' => '1', + 'name' => 'done', + 'report' => NULL, + 'plugin' => NULL, + 'segment' => '', + ), + ), + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-08', + 'date2' => '2018-03-08', + 'period' => '1', + 'name' => 'done5f4f9bafeda3443c3c2d4b2ef4dffadc', + 'report' => NULL, + 'plugin' => NULL, + 'segment' => 'browserCode==IE', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-07', + 'date2' => '2018-03-07', + 'period' => '1', + 'name' => 'done5f4f9bafeda3443c3c2d4b2ef4dffadc', + 'report' => NULL, + 'plugin' => NULL, + 'segment' => 'browserCode==IE', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-06', + 'date2' => '2018-03-06', + 'period' => '1', + 'name' => 'done.Actions', + 'report' => 'testReport', + 'plugin' => 'Actions', + 'segment' => '', + ), + ), + array ( + array ( // duplicate, processed but if in progress or recent should be skipped + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-06', + 'date2' => '2018-03-06', + 'period' => '1', + 'name' => 'done', + 'report' => NULL, + 'plugin' => NULL, + 'segment' => '', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-04', + 'date2' => '2018-03-04', + 'period' => '1', + 'name' => 'done', + 'report' => NULL, + 'plugin' => NULL, + 'segment' => '', + ), + ), + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-06', + 'date2' => '2018-03-06', + 'period' => '1', + 'name' => 'done5f4f9bafeda3443c3c2d4b2ef4dffadc', + 'report' => NULL, + 'plugin' => NULL, + 'segment' => 'browserCode==IE', + ), + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-04', + 'date2' => '2018-03-04', + 'period' => '1', + 'name' => 'done5f4f9bafeda3443c3c2d4b2ef4dffadc', + 'report' => NULL, + 'plugin' => NULL, + 'segment' => 'browserCode==IE', + ), + ), + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-04', + 'date2' => '2018-03-11', + 'period' => '2', + 'name' => 'done', + 'report' => NULL, + 'plugin' => NULL, + 'segment' => '', + ), + ), + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-04', + 'date2' => '2018-03-11', + 'period' => '2', + 'name' => 'done.Actions', + 'report' => 'testReport', + 'plugin' => 'Actions', + 'segment' => '', + ), + ), + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-04', + 'date2' => '2018-03-11', + 'period' => '2', + 'name' => 'done5f4f9bafeda3443c3c2d4b2ef4dffadc', + 'report' => NULL, + 'plugin' => NULL, + 'segment' => 'browserCode==IE', + ), + ), + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-01', + 'date2' => '2018-03-31', + 'period' => '3', + 'name' => 'done', + 'report' => NULL, + 'plugin' => NULL, + 'segment' => '', + ), + ), + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-01', + 'date2' => '2018-03-31', + 'period' => '3', + 'name' => 'done.Actions', + 'report' => 'testReport', + 'plugin' => 'Actions', + 'segment' => '', + ), + ), + array ( + array ( + 'idarchive' => '1', + 'idsite' => '1', + 'date1' => '2018-03-01', + 'date2' => '2018-03-31', + 'period' => '3', + 'name' => 'done5f4f9bafeda3443c3c2d4b2ef4dffadc', + 'report' => NULL, + 'plugin' => NULL, + 'segment' => 'browserCode==IE', + ), + ), + array ( // end of idsite=1 + ), + array ( + array ( + 'idarchive' => '1', + 'idsite' => '2', + 'date1' => '2020-03-30', + 'date2' => '2020-04-05', + 'period' => '2', + 'name' => 'done', + 'report' => 'testReport', + 'plugin' => NULL, + 'segment' => '', + ), + ), + array ( // end of idsite=2 + ), + array ( // end of idsite=3 + ), + ]; + + try { + $this->assertEquals($expectedInvalidationsFound, $iteratedInvalidations); + } catch (\Exception $ex) { + print "\nInvalidations inserted:\n" . var_export($invalidations, true) . "\n"; + throw $ex; + } + } + + private function makeTestArchiveFilter($restrictToDateRange = null, $restrictToPeriods = null, $segmentsToForce = null, $disableSegmentsArchiving = false) + { + $archiveFilter = new CronArchive\ArchiveFilter(); + if ($restrictToDateRange) { + $archiveFilter->setRestrictToDateRange(); + } + $archiveFilter->setDisableSegmentsArchiving($disableSegmentsArchiving); + if ($restrictToPeriods) { + $archiveFilter->setRestrictToPeriods($restrictToPeriods); + } + if ($segmentsToForce) { + $archiveFilter->setSegmentsToForceFromSegmentIds($segmentsToForce); + } + return $archiveFilter; + } + + private function insertInvalidations(array $invalidations) + { + $table = Common::prefixTable('archive_invalidations'); + foreach ($invalidations as $inv) { + $bind = [ + $inv['idarchive'], + $inv['name'], + $inv['idsite'], + $inv['date1'], + $inv['date2'], + $inv['period'], + $inv['report'], + ]; + Db::query("INSERT INTO `$table` (idarchive, name, idsite, date1, date2, period, ts_invalidated, report, status) + VALUES (?, ?, ?, ?, ?, ?, NOW(), ?, 0)", $bind); + } + } + + public function test_canSkipArchiveBecauseNoPoint_returnsTrueIfDateRangeHasNoVisits() + { + Fixture::createWebsite('2010-04-06'); + + Date::$now = strtotime('2020-04-05'); + + $cronArchive = new CronArchive(); + + $archiveFilter = $this->makeTestArchiveFilter(); + + $queueConsumer = new QueueConsumer( + StaticContainer::get(LoggerInterface::class), + new FixedSiteIds([1]), + 3, + 24, + new Model(), + new SegmentArchiving('beginning_of_time'), + $cronArchive, + new RequestParser(true), + $archiveFilter + ); + + $invalidation = [ + 'idsite' => 1, + 'period' => 1, + 'date1' => '2020-04-05', + 'date2' => '2020-04-05', + 'name' => 'done', + 'segment' => '', + ]; + + $result = $queueConsumer->canSkipArchiveBecauseNoPoint($invalidation); + $this->assertTrue($result); + } + + public function test_canSkipArchiveBecauseNoPoint_returnsFalseIfDateRangeHasVisits_AndPeriodDoesNotIncludeToday() + { + $idSite = Fixture::createWebsite('2015-02-03'); + + Date::$now = strtotime('2020-04-05'); + + $t = Fixture::getTracker($idSite, '2020-03-05 10:34:00'); + $t->setUrl('http://whatever.com'); + Fixture::checkResponse($t->doTrackPageView('test title')); + + $cronArchive = new CronArchive(); + + $archiveFilter = $this->makeTestArchiveFilter(); + + $queueConsumer = new QueueConsumer( + StaticContainer::get(LoggerInterface::class), + new FixedSiteIds([1]), + 3, + 24, + new Model(), + new SegmentArchiving('beginning_of_time'), + $cronArchive, + new RequestParser(true), + $archiveFilter + ); + + $invalidation = [ + 'idsite' => 1, + 'period' => 1, + 'date1' => '2020-03-05', + 'date2' => '2020-03-05', + 'name' => 'done', + 'segment' => '', + ]; + + $result = $queueConsumer->canSkipArchiveBecauseNoPoint($invalidation); + $this->assertFalse($result); + } + + public function test_usableArchiveExists_returnsTrueIfDateRangeHasVisits_AndPeriodIncludesToday_AndExistingArchiveIsRecent() + { + $idSite = Fixture::createWebsite('2015-02-03'); + + Date::$now = strtotime('2020-04-05'); + + $t = Fixture::getTracker($idSite, '2020-04-05 10:34:00'); + $t->setUrl('http://whatever.com'); + Fixture::checkResponse($t->doTrackPageView('test title')); + + $cronArchive = new CronArchive(); + + $archiveFilter = $this->makeTestArchiveFilter(); + + $queueConsumer = new QueueConsumer( + StaticContainer::get(LoggerInterface::class), + new FixedSiteIds([1]), + 3, + 24, + new Model(), + new SegmentArchiving('beginning_of_time'), + $cronArchive, + new RequestParser(true), + $archiveFilter + ); + + $invalidation = [ + 'idsite' => 1, + 'period' => 2, + 'date1' => '2020-03-30', + 'date2' => '2020-04-05', + 'name' => 'done', + 'segment' => '', + ]; + + $tsArchived = Date::factory('now')->subSeconds(100)->getDatetime(); + + $archiveTable = ArchiveTableCreator::getNumericTable(Date::factory('2020-03-30')); + Db::query("INSERT INTO $archiveTable (idarchive, idsite, period, date1, date2, name, value, ts_archived) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", [ + 1, 1,2, '2020-03-30', '2020-04-05', 'done', ArchiveWriter::DONE_INVALIDATED, $tsArchived + ]); + + $result = $queueConsumer->usableArchiveExists($invalidation); + $this->assertTrue($result); + } + + public function test_canSkipArchiveBecauseNoPoint_returnsFalseIfDateRangeHasVisits_AndPeriodIncludesToday_AndOnlyExistingArchiveIsRecentButPartial() + { + $idSite = Fixture::createWebsite('2015-02-03'); + + Date::$now = strtotime('2020-04-05'); + + $t = Fixture::getTracker($idSite, '2020-04-05 10:34:00'); + $t->setUrl('http://whatever.com'); + Fixture::checkResponse($t->doTrackPageView('test title')); + + $cronArchive = new CronArchive(); + + $archiveFilter = $this->makeTestArchiveFilter(); + + $queueConsumer = new QueueConsumer( + StaticContainer::get(LoggerInterface::class), + new FixedSiteIds([1]), + 3, + 24, + new Model(), + new SegmentArchiving('beginning_of_time'), + $cronArchive, + new RequestParser(true), + $archiveFilter + ); + + $invalidation = [ + 'idsite' => 1, + 'period' => 2, + 'date1' => '2020-03-30', + 'date2' => '2020-04-05', + 'name' => 'done', + 'segment' => '', + ]; + + $tsArchived = Date::factory('now')->subSeconds(100)->getDatetime(); + + $archiveTable = ArchiveTableCreator::getNumericTable(Date::factory('2020-03-30')); + Db::query("INSERT INTO $archiveTable (idarchive, idsite, period, date1, date2, name, value, ts_archived) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", [ + 1, 1,2, '2020-03-30', '2020-04-05', 'done', ArchiveWriter::DONE_PARTIAL, $tsArchived + ]); + + $result = $queueConsumer->canSkipArchiveBecauseNoPoint($invalidation); + $this->assertFalse($result); + } + + protected static function configureFixture($fixture) + { + parent::configureFixture($fixture); + $fixture->createSuperUser = true; + } +}
\ No newline at end of file diff --git a/tests/PHPUnit/Integration/CronArchiveTest.php b/tests/PHPUnit/Integration/CronArchiveTest.php index 4a18ddb827..0fc0992f4f 100644 --- a/tests/PHPUnit/Integration/CronArchiveTest.php +++ b/tests/PHPUnit/Integration/CronArchiveTest.php @@ -8,18 +8,24 @@ namespace Piwik\Tests\Integration; +use Piwik\ArchiveProcessor\Parameters; use Piwik\Container\StaticContainer; use Piwik\CronArchive; use Piwik\DataAccess\ArchiveTableCreator; +use Piwik\DataAccess\ArchiveWriter; use Piwik\Date; use Piwik\Db; +use Piwik\Period\Factory; use Piwik\Plugins\CoreAdminHome\tests\Framework\Mock\API; use Piwik\Plugins\SegmentEditor\Model; +use Piwik\Segment; +use Piwik\Site; use Piwik\Tests\Framework\Fixture; use Piwik\Tests\Framework\Mock\FakeLogger; use Piwik\Tests\Framework\TestCase\IntegrationTestCase; use Piwik\Plugins\SegmentEditor\API as SegmentAPI; use Piwik\Version; +use Psr\Log\NullLogger; /** * @group Archiver @@ -27,6 +33,111 @@ use Piwik\Version; */ class CronArchiveTest extends IntegrationTestCase { + public function test_isThereExistingValidPeriod_returnsTrueIfPeriodHasToday_AndExistingArchiveIsNewEnough() + { + Fixture::createWebsite('2019-04-04 03:45:45'); + + Date::$now = strtotime('2020-04-05'); + + $archiver = new CronArchive(); + + $params = new Parameters(new Site(1), Factory::build('week', '2020-04-05'), new Segment('', [1])); + + $tsArchived = Date::now()->subSeconds(100)->getDatetime(); + + $archiveTable = ArchiveTableCreator::getNumericTable(Date::factory('2020-03-30')); + Db::query("INSERT INTO $archiveTable (idarchive, idsite, period, date1, date2, name, value, ts_archived) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", [ + 1, 1,2, '2020-03-30', '2020-04-05', 'done', ArchiveWriter::DONE_OK, $tsArchived + ]); + + $actual =$archiver->isThereExistingValidPeriod($params, $isYesterday = false); + $this->assertTrue($actual); + } + + public function test_isThereExistingValidPeriod_returnsTrueIfPeriodHasToday_AndExistingArchiveIsNewEnoughAndInvalidated() + { + Fixture::createWebsite('2019-04-04 03:45:45'); + + Date::$now = strtotime('2020-04-05'); + + $archiver = new CronArchive(); + + $params = new Parameters(new Site(1), Factory::build('week', '2020-04-05'), new Segment('', [1])); + + $tsArchived = Date::now()->subSeconds(100)->getDatetime(); + + $archiveTable = ArchiveTableCreator::getNumericTable(Date::factory('2020-03-30')); + Db::query("INSERT INTO $archiveTable (idarchive, idsite, period, date1, date2, name, value, ts_archived) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", [ + 1, 1,2, '2020-03-30', '2020-04-05', 'done', ArchiveWriter::DONE_INVALIDATED, $tsArchived + ]); + + $actual =$archiver->isThereExistingValidPeriod($params, $isYesterday = false); + $this->assertTrue($actual); + } + + public function test_isThereExistingValidPeriod_returnsTrueIfPeriodDoesNotHaveToday_AndExistingArchiveIsOk() + { + Fixture::createWebsite('2019-04-04 03:45:45'); + + Date::$now = strtotime('2020-04-05'); + + $archiver = new CronArchive(); + + $params = new Parameters(new Site(1), Factory::build('day', '2020-03-05'), new Segment('', [1])); + + $tsArchived = Date::now()->subDay(1)->getDatetime(); + + $archiveTable = ArchiveTableCreator::getNumericTable(Date::factory('2020-03-05')); + Db::query("INSERT INTO $archiveTable (idarchive, idsite, period, date1, date2, name, value, ts_archived) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", [ + 1, 1, 1, '2020-03-05', '2020-03-05', 'done', ArchiveWriter::DONE_OK, $tsArchived + ]); + + $actual =$archiver->isThereExistingValidPeriod($params, $isYesterday = false); + $this->assertTrue($actual); + } + + public function test_isThereExistingValidPeriod_returnsFalseIfDayHasChangedAndDateIsYesterday() + { + Fixture::createWebsite('2019-04-04 03:45:45'); + + Date::$now = strtotime('2020-04-05'); + + $archiver = new CronArchive(); + + $params = new Parameters(new Site(1), Factory::build('day', '2020-04-04'), new Segment('', [1])); + + $tsArchived = Date::now()->subDay(1)->getDatetime(); + + $archiveTable = ArchiveTableCreator::getNumericTable(Date::factory('2020-04-04')); + Db::query("INSERT INTO $archiveTable (idarchive, idsite, period, date1, date2, name, value, ts_archived) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", [ + 1, 1, 1, '2020-04-04', '2020-04-04', 'done', ArchiveWriter::DONE_OK, $tsArchived + ]); + + $actual =$archiver->isThereExistingValidPeriod($params, $isYesterday = true); + $this->assertFalse($actual); + } + + public function test_isThereExistingValidPeriod_returnsTrueIfDayHasNotChangedAndDateIsYesterday() + { + Fixture::createWebsite('2019-04-04 03:45:45'); + + Date::$now = strtotime('2020-04-05 06:23:40'); + + $archiver = new CronArchive(); + + $params = new Parameters(new Site(1), Factory::build('day', '2020-04-04'), new Segment('', [1])); + + $tsArchived = Date::now()->subSeconds(1500)->getDatetime(); + + $archiveTable = ArchiveTableCreator::getNumericTable(Date::factory('2020-04-04')); + Db::query("INSERT INTO $archiveTable (idarchive, idsite, period, date1, date2, name, value, ts_archived) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", [ + 1, 1, 1, '2020-04-04', '2020-04-04', 'done', ArchiveWriter::DONE_OK, $tsArchived + ]); + + $actual = $archiver->isThereExistingValidPeriod($params, $isYesterday = true); + $this->assertTrue($actual); + } + public function test_getColumnNamesFromTable() { Fixture::createWebsite('2014-12-12 00:01:02'); @@ -173,108 +284,135 @@ Checking for queued invalidations... Will invalidate archived reports for 2019-12-11 for following websites ids: 1 Will invalidate archived reports for 2019-12-10 for following websites ids: 1 Will invalidate archived reports for 2019-12-02 for following websites ids: 1 - Today archive can be skipped due to no visits, skipping invalidation... - Yesterday archive can be skipped due to no visits, skipping invalidation... + Today archive can be skipped due to no visits for idSite = 1, skipping invalidation... + Yesterday archive can be skipped due to no visits for idSite = 1, skipping invalidation... Segment "actions>=2" was created or changed recently and will therefore archive today (for site ID = 1) Segment "actions>=4" was created or changed recently and will therefore archive today (for site ID = 1) Done invalidating Start processing archives for site 1. -Found invalidated archive we can skip (no visits or latest archive is not invalidated). [idSite = 1, dates = 2020-01-01 - 2020-01-01, segment = actions>=2] -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Found invalidated archive we can skip (no visits or latest archive is not invalidated). [idSite = 1, dates = 2020-01-01 - 2020-01-31, segment = actions>=2] -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Found invalidated archive we can skip (no visits or latest archive is not invalidated). [idSite = 1, dates = 2020-01-01 - 2020-12-31, segment = actions>=2] -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Found invalidated archive we can skip (no visits or latest archive is not invalidated). [idSite = 1, dates = 2019-12-31 - 2019-12-31, segment = actions>=2] -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Found invalidated archive we can skip (no visits or latest archive is not invalidated). [idSite = 1, dates = 2019-12-30 - 2019-12-30, segment = actions>=2] -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Found invalidated archive we can skip (no visits or latest archive is not invalidated). [idSite = 1, dates = 2019-12-30 - 2020-01-05, segment = actions>=2] -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Found invalidated archive we can skip (no visits or latest archive is not invalidated). [idSite = 1, dates = 2019-12-23 - 2019-12-23, segment = actions>=2] -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Found invalidated archive we can skip (no visits or latest archive is not invalidated). [idSite = 1, dates = 2019-12-23 - 2019-12-29, segment = actions>=2] -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Found invalidated archive we can skip (no visits or latest archive is not invalidated). [idSite = 1, dates = 2019-12-16 - 2019-12-16, segment = actions>=2] -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Found invalidated archive we can skip (no visits or latest archive is not invalidated). [idSite = 1, dates = 2019-12-16 - 2019-12-22, segment = actions>=2] -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment '' is not in --force-idsegments -Found invalidated archive we can skip (no visits or latest archive is not invalidated). [idSite = 1, dates = 2019-12-09 - 2019-12-09, segment = actions>=2] -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Found archive with different period than others in concurrent batch, skipping until next batch: 1 -Found archive with different period than others in concurrent batch, skipping until next batch: 1 -Found archive with different period than others in concurrent batch, skipping until next batch: 1 -Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: done -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Found archive with different period than others in concurrent batch, skipping until next batch: 3 -Found archive with different period than others in concurrent batch, skipping until next batch: 3 -Found archive with different period than others in concurrent batch, skipping until next batch: 3 -Found archive with different period than others in concurrent batch, skipping until next batch: 3 -Found archive with different period than others in concurrent batch, skipping until next batch: 3 -Found archive with different period than others in concurrent batch, skipping until next batch: 3 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 +Found invalidated archive we can skip (no visits): [idinvalidation = 43, idsite = 1, period = day(2020-02-03 - 2020-02-03), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 73, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 73, idsite = 1, period = day(2020-02-03 - 2020-02-03), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 44, idsite = 1, period = week(2020-02-03 - 2020-02-09), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 74, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 74, idsite = 1, period = week(2020-02-03 - 2020-02-09), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 42, idsite = 1, period = day(2020-02-02 - 2020-02-02), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 72, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 72, idsite = 1, period = day(2020-02-02 - 2020-02-02), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 40, idsite = 1, period = day(2020-02-01 - 2020-02-01), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 70, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 70, idsite = 1, period = day(2020-02-01 - 2020-02-01), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 41, idsite = 1, period = month(2020-02-01 - 2020-02-29), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 71, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 71, idsite = 1, period = month(2020-02-01 - 2020-02-29), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 36, idsite = 1, period = week(2020-01-27 - 2020-02-02), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 66, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 66, idsite = 1, period = week(2020-01-27 - 2020-02-02), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 33, idsite = 1, period = day(2020-01-01 - 2020-01-01), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 63, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 63, idsite = 1, period = day(2020-01-01 - 2020-01-01), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 34, idsite = 1, period = month(2020-01-01 - 2020-01-31), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 64, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 64, idsite = 1, period = month(2020-01-01 - 2020-01-31), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 35, idsite = 1, period = year(2020-01-01 - 2020-12-31), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 65, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 65, idsite = 1, period = year(2020-01-01 - 2020-12-31), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 28, idsite = 1, period = day(2019-12-31 - 2019-12-31), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 58, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 58, idsite = 1, period = day(2019-12-31 - 2019-12-31), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 26, idsite = 1, period = day(2019-12-30 - 2019-12-30), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 56, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 56, idsite = 1, period = day(2019-12-30 - 2019-12-30), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 27, idsite = 1, period = week(2019-12-30 - 2020-01-05), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 57, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 57, idsite = 1, period = week(2019-12-30 - 2020-01-05), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 24, idsite = 1, period = day(2019-12-23 - 2019-12-23), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 54, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 54, idsite = 1, period = day(2019-12-23 - 2019-12-23), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 25, idsite = 1, period = week(2019-12-23 - 2019-12-29), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 55, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 55, idsite = 1, period = week(2019-12-23 - 2019-12-29), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 22, idsite = 1, period = day(2019-12-16 - 2019-12-16), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 52, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 52, idsite = 1, period = day(2019-12-16 - 2019-12-16), name = done49a9440bd6dba4b8850035e09d043c67] +Found invalidated archive we can skip (no visits): [idinvalidation = 23, idsite = 1, period = week(2019-12-16 - 2019-12-22), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 53, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 53, idsite = 1, period = week(2019-12-16 - 2019-12-22), name = done49a9440bd6dba4b8850035e09d043c67] +Skipping invalidated archive 1, segment '' is not in --force-idsegments: [idinvalidation = 1, idsite = 1, period = day(2019-12-12 - 2019-12-12), name = done] +Skipping invalidated archive 5, segment '' is not in --force-idsegments: [idinvalidation = 5, idsite = 1, period = day(2019-12-11 - 2019-12-11), name = done] +Skipping invalidated archive 9, segment '' is not in --force-idsegments: [idinvalidation = 9, idsite = 1, period = day(2019-12-10 - 2019-12-10), name = done] +Found invalidated archive we can skip (no visits): [idinvalidation = 20, idsite = 1, period = day(2019-12-09 - 2019-12-09), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Skipping invalidated archive 50, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 50, idsite = 1, period = day(2019-12-09 - 2019-12-09), name = done49a9440bd6dba4b8850035e09d043c67] +Skipping invalidated archive 2, segment '' is not in --force-idsegments: [idinvalidation = 2, idsite = 1, period = week(2019-12-09 - 2019-12-15), name = done] +Skipping invalidated archive 6, segment '' is not in --force-idsegments: [idinvalidation = 6, idsite = 1, period = week(2019-12-09 - 2019-12-15), name = done] +Skipping invalidated archive 10, segment '' is not in --force-idsegments: [idinvalidation = 10, idsite = 1, period = week(2019-12-09 - 2019-12-15), name = done] +Processing invalidation: [idinvalidation = 21, idsite = 1, period = week(2019-12-09 - 2019-12-15), name = donee0512c03f7c20af6ef96a8d792c6bb9f]. +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 51, idsite = 1, period = week(2019-12-09 - 2019-12-15), name = done49a9440bd6dba4b8850035e09d043c67] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 13, idsite = 1, period = day(2019-12-02 - 2019-12-02), name = done] +Processing invalidation: [idinvalidation = 17, idsite = 1, period = day(2019-12-02 - 2019-12-02), name = donee0512c03f7c20af6ef96a8d792c6bb9f]. +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 47, idsite = 1, period = day(2019-12-02 - 2019-12-02), name = done49a9440bd6dba4b8850035e09d043c67] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 14, idsite = 1, period = week(2019-12-02 - 2019-12-08), name = done] +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 18, idsite = 1, period = week(2019-12-02 - 2019-12-08), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 48, idsite = 1, period = week(2019-12-02 - 2019-12-08), name = done49a9440bd6dba4b8850035e09d043c67] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 3, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 7, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 11, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 15, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done] +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 19, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 49, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done49a9440bd6dba4b8850035e09d043c67] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 4, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 8, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 12, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 16, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 32, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 62, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done49a9440bd6dba4b8850035e09d043c67] No next invalidated archive. -Starting archiving for ?module=API&method=API.get&idSite=1&period=week&date=2019-12-09&format=json&segment=actions%3E%3D2&trigger=archivephp -Starting archiving for ?module=API&method=API.get&idSite=1&period=week&date=2019-12-02&format=json&segment=actions%3E%3D2&trigger=archivephp +Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=1&period=week&date=2019-12-09&format=json&segment=actions%3E%3D2&trigger=archivephp +Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=1&period=day&date=2019-12-02&format=json&segment=actions%3E%3D2&trigger=archivephp Archived website id 1, period = week, date = 2019-12-09, segment = 'actions%3E%3D2', 0 visits found. Time elapsed: %fs -Archived website id 1, period = week, date = 2019-12-02, segment = 'actions%3E%3D2', 0 visits found. Time elapsed: %fs -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Found archive with different period than others in concurrent batch, skipping until next batch: 2 -Found archive with different period than others in concurrent batch, skipping until next batch: 3 -Found archive with different period than others in concurrent batch, skipping until next batch: 3 -Found archive with different period than others in concurrent batch, skipping until next batch: 3 -Found archive with different period than others in concurrent batch, skipping until next batch: 3 -Found archive with different period than others in concurrent batch, skipping until next batch: 3 -Found archive with different period than others in concurrent batch, skipping until next batch: 3 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -No next invalidated archive. -Starting archiving for ?module=API&method=API.get&idSite=1&period=day&date=2019-12-02&format=json&segment=actions%3E%3D2&trigger=archivephp Archived website id 1, period = day, date = 2019-12-02, segment = 'actions%3E%3D2', 0 visits found. Time elapsed: %fs -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 -Found archive with different period than others in concurrent batch, skipping until next batch: 4 +Skipping invalidated archive 51, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 51, idsite = 1, period = week(2019-12-09 - 2019-12-15), name = done49a9440bd6dba4b8850035e09d043c67] +Skipping invalidated archive 13, segment '' is not in --force-idsegments: [idinvalidation = 13, idsite = 1, period = day(2019-12-02 - 2019-12-02), name = done] +Skipping invalidated archive 47, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 47, idsite = 1, period = day(2019-12-02 - 2019-12-02), name = done49a9440bd6dba4b8850035e09d043c67] +Skipping invalidated archive 14, segment '' is not in --force-idsegments: [idinvalidation = 14, idsite = 1, period = week(2019-12-02 - 2019-12-08), name = done] +Processing invalidation: [idinvalidation = 18, idsite = 1, period = week(2019-12-02 - 2019-12-08), name = donee0512c03f7c20af6ef96a8d792c6bb9f]. +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 48, idsite = 1, period = week(2019-12-02 - 2019-12-08), name = done49a9440bd6dba4b8850035e09d043c67] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 3, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 7, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 11, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 15, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done] +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 19, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 49, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done49a9440bd6dba4b8850035e09d043c67] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 4, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 8, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 12, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 16, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 32, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 62, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done49a9440bd6dba4b8850035e09d043c67] +No next invalidated archive. +Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=1&period=week&date=2019-12-02&format=json&segment=actions%3E%3D2&trigger=archivephp +Archived website id 1, period = week, date = 2019-12-02, segment = 'actions%3E%3D2', 0 visits found. Time elapsed: %fs +Skipping invalidated archive 48, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 48, idsite = 1, period = week(2019-12-02 - 2019-12-08), name = done49a9440bd6dba4b8850035e09d043c67] +Skipping invalidated archive 3, segment '' is not in --force-idsegments: [idinvalidation = 3, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done] +Skipping invalidated archive 7, segment '' is not in --force-idsegments: [idinvalidation = 7, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done] +Skipping invalidated archive 11, segment '' is not in --force-idsegments: [idinvalidation = 11, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done] +Skipping invalidated archive 15, segment '' is not in --force-idsegments: [idinvalidation = 15, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done] +Processing invalidation: [idinvalidation = 19, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = donee0512c03f7c20af6ef96a8d792c6bb9f]. +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 49, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done49a9440bd6dba4b8850035e09d043c67] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 4, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 8, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 12, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Found archive with different done flag type (segment vs. no segment) in concurrent batch, skipping until next batch: [idinvalidation = 16, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 32, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = donee0512c03f7c20af6ef96a8d792c6bb9f] +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 62, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done49a9440bd6dba4b8850035e09d043c67] No next invalidated archive. -Starting archiving for ?module=API&method=API.get&idSite=1&period=month&date=2019-12-01&format=json&segment=actions%3E%3D2&trigger=archivephp +Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=1&period=month&date=2019-12-01&format=json&segment=actions%3E%3D2&trigger=archivephp Archived website id 1, period = month, date = 2019-12-01, segment = 'actions%3E%3D2', 0 visits found. Time elapsed: %fs -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment '' is not in --force-idsegments -Skipping invalidated archive : segment 'actions>=4' is not in --force-idsegments +Skipping invalidated archive 49, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 49, idsite = 1, period = month(2019-12-01 - 2019-12-31), name = done49a9440bd6dba4b8850035e09d043c67] +Skipping invalidated archive 4, segment '' is not in --force-idsegments: [idinvalidation = 4, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Skipping invalidated archive 8, segment '' is not in --force-idsegments: [idinvalidation = 8, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Skipping invalidated archive 12, segment '' is not in --force-idsegments: [idinvalidation = 12, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Skipping invalidated archive 16, segment '' is not in --force-idsegments: [idinvalidation = 16, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done] +Processing invalidation: [idinvalidation = 32, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = donee0512c03f7c20af6ef96a8d792c6bb9f]. +Found archive with intersecting period with others in concurrent batch, skipping until next batch: [idinvalidation = 62, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done49a9440bd6dba4b8850035e09d043c67] No next invalidated archive. -Starting archiving for ?module=API&method=API.get&idSite=1&period=year&date=2019-01-01&format=json&segment=actions%3E%3D2&trigger=archivephp +Starting archiving for ?module=API&method=CoreAdminHome.archiveReports&idSite=1&period=year&date=2019-01-01&format=json&segment=actions%3E%3D2&trigger=archivephp Archived website id 1, period = year, date = 2019-01-01, segment = 'actions%3E%3D2', 0 visits found. Time elapsed: %fs +Skipping invalidated archive 62, segment 'actions>=4' is not in --force-idsegments: [idinvalidation = 62, idsite = 1, period = year(2019-01-01 - 2019-12-31), name = done49a9440bd6dba4b8850035e09d043c67] No next invalidated archive. Finished archiving for site 1, 5 API requests, Time elapsed: %fs [1 / 1 done] No more sites left to archive, stopping. - +Done archiving! +--------------------------- +SUMMARY +Processed 5 archives. +Total API requests: 5 +done: 5 req, %d ms, no error +Time elapsed: %fs LOG; $this->assertStringMatchesFormat($expected, $logger->output); @@ -302,8 +440,8 @@ LOG; START Starting Matomo reports archiving... Checking for queued invalidations... - Today archive can be skipped due to no visits, skipping invalidation... - Yesterday archive can be skipped due to no visits, skipping invalidation... + Today archive can be skipped due to no visits for idSite = 1, skipping invalidation... + Yesterday archive can be skipped due to no visits for idSite = 1, skipping invalidation... Done invalidating Start processing archives for site 1. No next invalidated archive. diff --git a/tests/PHPUnit/Integration/DataAccess/ArchiveInvalidatorTest.php b/tests/PHPUnit/Integration/DataAccess/ArchiveInvalidatorTest.php index dedce82e58..e49a2c9699 100644 --- a/tests/PHPUnit/Integration/DataAccess/ArchiveInvalidatorTest.php +++ b/tests/PHPUnit/Integration/DataAccess/ArchiveInvalidatorTest.php @@ -11,6 +11,7 @@ namespace Piwik\Tests\Integration\DataAccess; use Piwik\ArchiveProcessor\ArchivingStatus; use Piwik\ArchiveProcessor\Rules; use Piwik\Common; +use Piwik\Config; use Piwik\Container\StaticContainer; use Piwik\DataAccess\ArchiveTableCreator; use Piwik\DataAccess\ArchiveWriter; @@ -21,6 +22,7 @@ use Piwik\Option; use Piwik\Piwik; use Piwik\Plugins\CoreAdminHome\Tasks\ArchivesToPurgeDistributedList; use Piwik\Plugins\PrivacyManager\PrivacyManager; +use Piwik\Plugins\SegmentEditor\API; use Piwik\Tests\Framework\Fixture; use Piwik\Tests\Framework\TestCase\IntegrationTestCase; use Piwik\Archive\ArchiveInvalidator; @@ -67,6 +69,10 @@ class ArchiveInvalidatorTest extends IntegrationTestCase for ($i = 0; $i != 10; ++$i) { Fixture::createWebsite('2012-03-04'); } + + self::addVisitToEachSite(); + + Option::deleteLike('%report_to_invalidate_%'); // test w/ a blank slate } public function setUp(): void @@ -314,6 +320,7 @@ class ArchiveInvalidatorTest extends IntegrationTestCase 'date2' => '2016-12-31', 'period' => '4', 'name' => 'done', + 'report' => null, ], [ 'idarchive' => null, @@ -322,6 +329,7 @@ class ArchiveInvalidatorTest extends IntegrationTestCase 'date2' => '2016-03-06', 'period' => '2', 'name' => 'done', + 'report' => null, ], [ 'idarchive' => null, @@ -330,6 +338,7 @@ class ArchiveInvalidatorTest extends IntegrationTestCase 'date2' => '2016-03-31', 'period' => '3', 'name' => 'done', + 'report' => null, ], [ 'idarchive' => null, @@ -338,6 +347,7 @@ class ArchiveInvalidatorTest extends IntegrationTestCase 'date2' => '2016-03-04', 'period' => '1', 'name' => 'done', + 'report' => null, ], ]; @@ -349,7 +359,7 @@ class ArchiveInvalidatorTest extends IntegrationTestCase * @dataProvider getTestDataForMarkArchivesAsInvalidated */ public function test_markArchivesAsInvalidated_MarksCorrectArchivesAsInvalidated($idSites, $dates, $period, $segment, $cascadeDown, $expectedIdArchives, - $expectedInvalidatedArchives) + $expectedInvalidatedArchives, $name = null) { $this->insertArchiveRowsForTest(); @@ -359,7 +369,8 @@ class ArchiveInvalidatorTest extends IntegrationTestCase /** @var ArchiveInvalidator $archiveInvalidator */ $archiveInvalidator = self::$fixture->piwikEnvironment->getContainer()->get('Piwik\Archive\ArchiveInvalidator'); - $result = $archiveInvalidator->markArchivesAsInvalidated($idSites, $dates, $period, $segment, $cascadeDown); + + $result = $archiveInvalidator->markArchivesAsInvalidated($idSites, $dates, $period, $segment, $cascadeDown, false, $name); $this->assertEquals($dates, $result->processedDates); @@ -367,11 +378,11 @@ class ArchiveInvalidatorTest extends IntegrationTestCase // Remove empty values (some new empty entries may be added each month) $idArchives = array_filter($idArchives); - $expectedIdArchives = array_filter($expectedIdArchives); $this->assertEquals($expectedIdArchives, $idArchives); $invalidatedIdArchives = $this->getInvalidatedArchiveTableEntries(); + $this->assertEqualsSorted($expectedInvalidatedArchives, $invalidatedIdArchives); } @@ -418,26 +429,26 @@ class ArchiveInvalidatorTest extends IntegrationTestCase ], ), [ - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2014-12-29', 'date2' => '2015-01-04', 'period' => '2', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-01', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-31', 'period' => '3', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-02-01', 'date2' => '2015-02-28', 'period' => '3', 'name' => 'done'], - ['idarchive' => '85', 'idsite' => '1', 'date1' => '2015-02-02', 'date2' => '2015-02-08', 'period' => '2', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-02-05', 'date2' => '2015-02-05', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-04-01', 'date2' => '2015-04-30', 'period' => '3', 'name' => 'done'], - ['idarchive' => '100', 'idsite' => '1', 'date1' => '2015-04-27', 'date2' => '2015-05-03', 'period' => '2', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-04-30', 'date2' => '2015-04-30', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2014-12-29', 'date2' => '2015-01-04', 'period' => '2', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-01-01', 'date2' => '2015-01-01', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-01-01', 'date2' => '2015-01-31', 'period' => '3', 'name' => 'done'], - ['idarchive' => '110', 'idsite' => '2', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-02-01', 'date2' => '2015-02-28', 'period' => '3', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-02-02', 'date2' => '2015-02-08', 'period' => '2', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-02-05', 'date2' => '2015-02-05', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-04-01', 'date2' => '2015-04-30', 'period' => '3', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-04-27', 'date2' => '2015-05-03', 'period' => '2', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-04-30', 'date2' => '2015-04-30', 'period' => '1', 'name' => 'done'], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2014-12-29', 'date2' => '2015-01-04', 'period' => '2', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-01', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-31', 'period' => '3', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-02-01', 'date2' => '2015-02-28', 'period' => '3', 'name' => 'done', 'report' => null], + ['idarchive' => '85', 'idsite' => '1', 'date1' => '2015-02-02', 'date2' => '2015-02-08', 'period' => '2', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-02-05', 'date2' => '2015-02-05', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-04-01', 'date2' => '2015-04-30', 'period' => '3', 'name' => 'done', 'report' => null], + ['idarchive' => '100', 'idsite' => '1', 'date1' => '2015-04-27', 'date2' => '2015-05-03', 'period' => '2', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-04-30', 'date2' => '2015-04-30', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2014-12-29', 'date2' => '2015-01-04', 'period' => '2', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-01-01', 'date2' => '2015-01-01', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-01-01', 'date2' => '2015-01-31', 'period' => '3', 'name' => 'done', 'report' => null], + ['idarchive' => '110', 'idsite' => '2', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-02-01', 'date2' => '2015-02-28', 'period' => '3', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-02-02', 'date2' => '2015-02-08', 'period' => '2', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-02-05', 'date2' => '2015-02-05', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-04-01', 'date2' => '2015-04-30', 'period' => '3', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-04-27', 'date2' => '2015-05-03', 'period' => '2', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '2', 'date1' => '2015-04-30', 'date2' => '2015-04-30', 'period' => '1', 'name' => 'done', 'report' => null], ], ), @@ -455,8 +466,8 @@ class ArchiveInvalidatorTest extends IntegrationTestCase ), ), [ - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-31', 'period' => '3', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done'], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-31', 'period' => '3', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done', 'report' => null], ], ), @@ -513,44 +524,44 @@ class ArchiveInvalidatorTest extends IntegrationTestCase ), ), [ - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2014-12-29', 'date2' => '2015-01-04', 'period' => '2', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-01', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-31', 'period' => '3', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-02', 'date2' => '2015-01-02', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-03', 'date2' => '2015-01-03', 'period' => '1', 'name' => 'done'], - ['idarchive' => '10', 'idsite' => '1', 'date1' => '2015-01-04', 'date2' => '2015-01-04', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-05', 'date2' => '2015-01-05', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-05', 'date2' => '2015-01-11', 'period' => '2', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-06', 'date2' => '2015-01-06', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-07', 'date2' => '2015-01-07', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-08', 'date2' => '2015-01-08', 'period' => '1', 'name' => 'done'], - ['idarchive' => '25', 'idsite' => '1', 'date1' => '2015-01-09', 'date2' => '2015-01-09', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-10', 'date2' => '2015-01-10', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-11', 'date2' => '2015-01-11', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-12', 'date2' => '2015-01-12', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-12', 'date2' => '2015-01-18', 'period' => '2', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-13', 'date2' => '2015-01-13', 'period' => '1', 'name' => 'done'], - ['idarchive' => '40', 'idsite' => '1', 'date1' => '2015-01-14', 'date2' => '2015-01-14', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-15', 'date2' => '2015-01-15', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-16', 'date2' => '2015-01-16', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-17', 'date2' => '2015-01-17', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-18', 'date2' => '2015-01-18', 'period' => '1', 'name' => 'done'], - ['idarchive' => '55', 'idsite' => '1', 'date1' => '2015-01-19', 'date2' => '2015-01-19', 'period' => '1', 'name' => 'done'], - ['idarchive' => '100', 'idsite' => '1', 'date1' => '2015-01-19', 'date2' => '2015-01-25', 'period' => '2', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-20', 'date2' => '2015-01-20', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-21', 'date2' => '2015-01-21', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-22', 'date2' => '2015-01-22', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-23', 'date2' => '2015-01-23', 'period' => '1', 'name' => 'done'], - ['idarchive' => '70', 'idsite' => '1', 'date1' => '2015-01-24', 'date2' => '2015-01-24', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-25', 'date2' => '2015-01-25', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-26', 'date2' => '2015-01-26', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-26', 'date2' => '2015-02-01', 'period' => '2', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-27', 'date2' => '2015-01-27', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-28', 'date2' => '2015-01-28', 'period' => '1', 'name' => 'done'], - ['idarchive' => '85', 'idsite' => '1', 'date1' => '2015-01-29', 'date2' => '2015-01-29', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-30', 'date2' => '2015-01-30', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-31', 'date2' => '2015-01-31', 'period' => '1', 'name' => 'done'], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2014-12-29', 'date2' => '2015-01-04', 'period' => '2', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-01', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-31', 'period' => '3', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-02', 'date2' => '2015-01-02', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-03', 'date2' => '2015-01-03', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => '10', 'idsite' => '1', 'date1' => '2015-01-04', 'date2' => '2015-01-04', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-05', 'date2' => '2015-01-05', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-05', 'date2' => '2015-01-11', 'period' => '2', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-06', 'date2' => '2015-01-06', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-07', 'date2' => '2015-01-07', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-08', 'date2' => '2015-01-08', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => '25', 'idsite' => '1', 'date1' => '2015-01-09', 'date2' => '2015-01-09', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-10', 'date2' => '2015-01-10', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-11', 'date2' => '2015-01-11', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-12', 'date2' => '2015-01-12', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-12', 'date2' => '2015-01-18', 'period' => '2', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-13', 'date2' => '2015-01-13', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => '40', 'idsite' => '1', 'date1' => '2015-01-14', 'date2' => '2015-01-14', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-15', 'date2' => '2015-01-15', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-16', 'date2' => '2015-01-16', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-17', 'date2' => '2015-01-17', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-18', 'date2' => '2015-01-18', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => '55', 'idsite' => '1', 'date1' => '2015-01-19', 'date2' => '2015-01-19', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => '100', 'idsite' => '1', 'date1' => '2015-01-19', 'date2' => '2015-01-25', 'period' => '2', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-20', 'date2' => '2015-01-20', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-21', 'date2' => '2015-01-21', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-22', 'date2' => '2015-01-22', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-23', 'date2' => '2015-01-23', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => '70', 'idsite' => '1', 'date1' => '2015-01-24', 'date2' => '2015-01-24', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-25', 'date2' => '2015-01-25', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-26', 'date2' => '2015-01-26', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-26', 'date2' => '2015-02-01', 'period' => '2', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-27', 'date2' => '2015-01-27', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-28', 'date2' => '2015-01-28', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => '85', 'idsite' => '1', 'date1' => '2015-01-29', 'date2' => '2015-01-29', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-30', 'date2' => '2015-01-30', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-31', 'date2' => '2015-01-31', 'period' => '1', 'name' => 'done', 'report' => null], ], ), @@ -595,27 +606,27 @@ class ArchiveInvalidatorTest extends IntegrationTestCase ], ), [ - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2014-01-01', 'date2' => '2014-12-31', 'period' => '4', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2014-12-01', 'date2' => '2014-12-31', 'period' => '3', 'name' => 'done'], - ['idarchive' => '85', 'idsite' => '1', 'date1' => '2014-12-29', 'date2' => '2014-12-29', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2014-12-29', 'date2' => '2015-01-04', 'period' => '2', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2014-12-30', 'date2' => '2014-12-30', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2014-12-31', 'date2' => '2014-12-31', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-01', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-31', 'period' => '3', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-02', 'date2' => '2015-01-02', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-03', 'date2' => '2015-01-03', 'period' => '1', 'name' => 'done'], - ['idarchive' => '10', 'idsite' => '1', 'date1' => '2015-01-04', 'date2' => '2015-01-04', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-26', 'date2' => '2015-01-26', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-26', 'date2' => '2015-02-01', 'period' => '2', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-27', 'date2' => '2015-01-27', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-28', 'date2' => '2015-01-28', 'period' => '1', 'name' => 'done'], - ['idarchive' => '85', 'idsite' => '1', 'date1' => '2015-01-29', 'date2' => '2015-01-29', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-30', 'date2' => '2015-01-30', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-31', 'date2' => '2015-01-31', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-02-01', 'date2' => '2015-02-01', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-02-01', 'date2' => '2015-02-28', 'period' => '3', 'name' => 'done'], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2014-01-01', 'date2' => '2014-12-31', 'period' => '4', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2014-12-01', 'date2' => '2014-12-31', 'period' => '3', 'name' => 'done', 'report' => null], + ['idarchive' => '85', 'idsite' => '1', 'date1' => '2014-12-29', 'date2' => '2014-12-29', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2014-12-29', 'date2' => '2015-01-04', 'period' => '2', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2014-12-30', 'date2' => '2014-12-30', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2014-12-31', 'date2' => '2014-12-31', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-01', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-31', 'period' => '3', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-02', 'date2' => '2015-01-02', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-03', 'date2' => '2015-01-03', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => '10', 'idsite' => '1', 'date1' => '2015-01-04', 'date2' => '2015-01-04', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-26', 'date2' => '2015-01-26', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-26', 'date2' => '2015-02-01', 'period' => '2', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-27', 'date2' => '2015-01-27', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-28', 'date2' => '2015-01-28', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => '85', 'idsite' => '1', 'date1' => '2015-01-29', 'date2' => '2015-01-29', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-30', 'date2' => '2015-01-30', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-31', 'date2' => '2015-01-31', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-02-01', 'date2' => '2015-02-01', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-02-01', 'date2' => '2015-02-28', 'period' => '3', 'name' => 'done', 'report' => null], ], ), @@ -688,44 +699,44 @@ class ArchiveInvalidatorTest extends IntegrationTestCase ), [ // TODO: super strange, there are two idarchive = 106 values here - ['idarchive' => '106', 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-31', 'period' => '3', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => '1', 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-01', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-02', 'date2' => '2015-01-02', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-03', 'date2' => '2015-01-03', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-04', 'date2' => '2015-01-04', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-05', 'date2' => '2015-01-11', 'period' => '2', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-05', 'date2' => '2015-01-05', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => '16', 'idsite' => '1', 'date1' => '2015-01-06', 'date2' => '2015-01-06', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-07', 'date2' => '2015-01-07', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-08', 'date2' => '2015-01-08', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-09', 'date2' => '2015-01-09', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-10', 'date2' => '2015-01-10', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => '31', 'idsite' => '1', 'date1' => '2015-01-11', 'date2' => '2015-01-11', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-12', 'date2' => '2015-01-18', 'period' => '2', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-12', 'date2' => '2015-01-12', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-13', 'date2' => '2015-01-13', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-14', 'date2' => '2015-01-14', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-15', 'date2' => '2015-01-15', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => '46', 'idsite' => '1', 'date1' => '2015-01-16', 'date2' => '2015-01-16', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-17', 'date2' => '2015-01-17', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-18', 'date2' => '2015-01-18', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-19', 'date2' => '2015-01-25', 'period' => '2', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-19', 'date2' => '2015-01-19', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-20', 'date2' => '2015-01-20', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => '61', 'idsite' => '1', 'date1' => '2015-01-21', 'date2' => '2015-01-21', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-22', 'date2' => '2015-01-22', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-23', 'date2' => '2015-01-23', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-24', 'date2' => '2015-01-24', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-25', 'date2' => '2015-01-25', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => '76', 'idsite' => '1', 'date1' => '2015-01-26', 'date2' => '2015-01-26', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-26', 'date2' => '2015-02-01', 'period' => '2', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-27', 'date2' => '2015-01-27', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-28', 'date2' => '2015-01-28', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-29', 'date2' => '2015-01-29', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-30', 'date2' => '2015-01-30', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => '106', 'idsite' => '1', 'date1' => '2014-12-29', 'date2' => '2015-01-04', 'period' => '2', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], - ['idarchive' => '91', 'idsite' => '1', 'date1' => '2015-01-31', 'date2' => '2015-01-31', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341'], + ['idarchive' => '106', 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-31', 'period' => '3', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => '1', 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-01', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-02', 'date2' => '2015-01-02', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-03', 'date2' => '2015-01-03', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-04', 'date2' => '2015-01-04', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-05', 'date2' => '2015-01-11', 'period' => '2', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-05', 'date2' => '2015-01-05', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => '16', 'idsite' => '1', 'date1' => '2015-01-06', 'date2' => '2015-01-06', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-07', 'date2' => '2015-01-07', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-08', 'date2' => '2015-01-08', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-09', 'date2' => '2015-01-09', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-10', 'date2' => '2015-01-10', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => '31', 'idsite' => '1', 'date1' => '2015-01-11', 'date2' => '2015-01-11', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-12', 'date2' => '2015-01-18', 'period' => '2', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-12', 'date2' => '2015-01-12', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-13', 'date2' => '2015-01-13', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-14', 'date2' => '2015-01-14', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-15', 'date2' => '2015-01-15', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => '46', 'idsite' => '1', 'date1' => '2015-01-16', 'date2' => '2015-01-16', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-17', 'date2' => '2015-01-17', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-18', 'date2' => '2015-01-18', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-19', 'date2' => '2015-01-25', 'period' => '2', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-19', 'date2' => '2015-01-19', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-20', 'date2' => '2015-01-20', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => '61', 'idsite' => '1', 'date1' => '2015-01-21', 'date2' => '2015-01-21', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-22', 'date2' => '2015-01-22', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-23', 'date2' => '2015-01-23', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-24', 'date2' => '2015-01-24', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-25', 'date2' => '2015-01-25', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => '76', 'idsite' => '1', 'date1' => '2015-01-26', 'date2' => '2015-01-26', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-26', 'date2' => '2015-02-01', 'period' => '2', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-27', 'date2' => '2015-01-27', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-28', 'date2' => '2015-01-28', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-29', 'date2' => '2015-01-29', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-30', 'date2' => '2015-01-30', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => '106', 'idsite' => '1', 'date1' => '2014-12-29', 'date2' => '2015-01-04', 'period' => '2', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], + ['idarchive' => '91', 'idsite' => '1', 'date1' => '2015-01-31', 'date2' => '2015-01-31', 'period' => '1', 'name' => 'done3736b708e4d20cfc10610e816a1b2341', 'report' => null], ], ), @@ -747,12 +758,30 @@ class ArchiveInvalidatorTest extends IntegrationTestCase ), ), [ - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-05-05', 'date2' => '2015-05-05', 'period' => '1', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-05-04', 'date2' => '2015-05-10', 'period' => '2', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-05-01', 'date2' => '2015-05-31', 'period' => '3', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done'], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-05-05', 'date2' => '2015-05-05', 'period' => '1', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-05-04', 'date2' => '2015-05-10', 'period' => '2', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-05-01', 'date2' => '2015-05-31', 'period' => '3', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done', 'report' => null], ], ), + + // period before site creation date + [ + [1], + ['2012-03-02'], + '', + null, + false, + [ + // empty + ], + [ + // month week and year exist, but not day since it is before the site was created + ['idarchive' => null, 'idsite' => 1, 'date1' => '2012-03-01', 'date2' => '2012-03-31', 'period' => 3, 'name' => 'done', 'report' => null], + ['idarchive' => null, 'idsite' => 1, 'date1' => '2012-02-27', 'date2' => '2012-03-04', 'period' => 2, 'name' => 'done', 'report' => null], + ['idarchive' => null, 'idsite' => 1, 'date1' => '2012-01-01', 'date2' => '2012-12-31', 'period' => 4, 'name' => 'done', 'report' => null], + ], + ], ); } @@ -916,14 +945,257 @@ class ArchiveInvalidatorTest extends IntegrationTestCase $this->assertEquals([], $archives); $expectedInvalidatedTableEntries = [ - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-03-04', 'date2' => '2015-03-06', 'period' => '5', 'name' => 'done'], - ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2016-04-03', 'date2' => '2016-05-12', 'period' => '5', 'name' => 'done'], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-03-04', 'date2' => '2015-03-06', 'period' => '5', 'name' => 'done', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2016-04-03', 'date2' => '2016-05-12', 'period' => '5', 'name' => 'done', 'report' => null], ]; $invalidatedTableEntries = $this->getInvalidatedArchiveTableEntries(); $this->assertEquals($expectedInvalidatedTableEntries, $invalidatedTableEntries); } + public function test_markArchivesAsInvalidated_invalidatesIndividualPluginNames() + { + $idSites = [1]; + $dates = ['2015-01-11']; + $period = 'day'; + $segment = new Segment('', [1]); + $cascadeDown = false; + $expectedIdArchives = []; + $expectedInvalidatedArchives = [ + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-31', 'period' => '3', 'name' => 'done.ExamplePlugin', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done.ExamplePlugin', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-05', 'date2' => '2015-01-11', 'period' => '2', 'name' => 'done.ExamplePlugin', 'report' => null], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-11', 'date2' => '2015-01-11', 'period' => '1', 'name' => 'done.ExamplePlugin', 'report' => null], + ]; + $plugin = 'ExamplePlugin'; + + $this->test_markArchivesAsInvalidated_MarksCorrectArchivesAsInvalidated($idSites, $dates, $period, $segment, $cascadeDown, $expectedIdArchives, + $expectedInvalidatedArchives, $plugin); + } + + public function test_markArchivesAsInvalidated_invalidatesIndividualReports() + { + $idSites = [1]; + $dates = ['2015-01-11']; + $period = 'day'; + $segment = new Segment('', [1]); + $cascadeDown = false; + $expectedIdArchives = []; + $expectedInvalidatedArchives = [ + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-01-31', 'period' => '3', 'name' => 'done.ExamplePlugin', 'report' => 'someReport'], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-01', 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done.ExamplePlugin', 'report' => 'someReport'], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-05', 'date2' => '2015-01-11', 'period' => '2', 'name' => 'done.ExamplePlugin', 'report' => 'someReport'], + ['idarchive' => NULL, 'idsite' => '1', 'date1' => '2015-01-11', 'date2' => '2015-01-11', 'period' => '1', 'name' => 'done.ExamplePlugin', 'report' => 'someReport'], + ]; + $report = 'ExamplePlugin.someReport'; + + $this->test_markArchivesAsInvalidated_MarksCorrectArchivesAsInvalidated($idSites, $dates, $period, $segment, $cascadeDown, $expectedIdArchives, + $expectedInvalidatedArchives, $report); + } + + public function test_reArchiveReport_createsCorrectInvalidationEntries_ifNoReportSpecified() + { + Date::$now = strtotime('2020-06-16 12:00:00'); + + Config::getInstance()->General['rearchive_reports_in_past_last_n_months'] = 'last1'; + + $this->invalidator->reArchiveReport([1], 'VisitsSummary'); + + $expectedInvalidations = [ + array ( + 'idsite' => '1', + 'period' => '1', + 'name' => 'done.VisitsSummary', + 'report' => NULL, + 'dates' => '2020-05-01,2020-05-01|2020-05-02,2020-05-02|2020-05-03,2020-05-03|2020-05-04,2020-05-04|2020-05-05,2020-05-05|2020-05-06,2020-05-06' + . '|2020-05-07,2020-05-07|2020-05-08,2020-05-08|2020-05-09,2020-05-09|2020-05-10,2020-05-10|2020-05-11,2020-05-11|2020-05-12,2020-05-12' + . '|2020-05-13,2020-05-13|2020-05-14,2020-05-14|2020-05-15,2020-05-15|2020-05-16,2020-05-16|2020-05-17,2020-05-17|2020-05-18,2020-05-18' + . '|2020-05-19,2020-05-19|2020-05-20,2020-05-20|2020-05-21,2020-05-21|2020-05-22,2020-05-22|2020-05-23,2020-05-23|2020-05-24,2020-05-24' + . '|2020-05-25,2020-05-25|2020-05-26,2020-05-26|2020-05-27,2020-05-27|2020-05-28,2020-05-28|2020-05-29,2020-05-29|2020-05-30,2020-05-30' + . '|2020-05-31,2020-05-31|2020-06-01,2020-06-01|2020-06-02,2020-06-02|2020-06-03,2020-06-03|2020-06-04,2020-06-04|2020-06-05,2020-06-05' + . '|2020-06-06,2020-06-06|2020-06-07,2020-06-07|2020-06-08,2020-06-08|2020-06-09,2020-06-09|2020-06-10,2020-06-10|2020-06-11,2020-06-11' + . '|2020-06-12,2020-06-12|2020-06-13,2020-06-13|2020-06-14,2020-06-14', + 'count' => '45', + ), + array ( + 'idsite' => '1', + 'period' => '2', + 'name' => 'done.VisitsSummary', + 'report' => NULL, + 'dates' => '2020-05-04,2020-05-10|2020-05-11,2020-05-17|2020-05-18,2020-05-24|2020-05-25,2020-05-31|2020-04-27,2020-05-03|2020-06-01,2020-06-07' + . '|2020-06-08,2020-06-14', + 'count' => '7', + ), + array ( + 'idsite' => '1', + 'period' => '3', + 'name' => 'done.VisitsSummary', + 'report' => NULL, + 'dates' => '2020-05-01,2020-05-31|2020-06-01,2020-06-30', + 'count' => '2', + ), + array ( + 'idsite' => '1', + 'period' => '4', + 'name' => 'done.VisitsSummary', + 'report' => NULL, + 'dates' => '2020-01-01,2020-12-31', + 'count' => '1', + ), + ]; + + $actualInvalidations = $this->getInvalidatedArchiveTableEntriesSummary(); + + $this->assertEquals($expectedInvalidations, $actualInvalidations); + } + + public function test_reArchiveReport_createsCorrectInvalidationEntries_ifReportSpecified() + { + Date::$now = strtotime('2020-06-16 12:00:00'); + + Config::getInstance()->General['rearchive_reports_in_past_last_n_months'] = 'last1'; + + $this->invalidator->reArchiveReport([1], 'VisitsSummary', 'some.Report'); + + $expectedInvalidations = [ + array ( + 'idsite' => '1', + 'period' => '1', + 'name' => 'done.VisitsSummary', + 'report' => 'some.Report', + 'dates' => '2020-05-01,2020-05-01|2020-05-02,2020-05-02|2020-05-03,2020-05-03|2020-05-04,2020-05-04|2020-05-05,2020-05-05' + . '|2020-05-06,2020-05-06|2020-05-07,2020-05-07|2020-05-08,2020-05-08|2020-05-09,2020-05-09|2020-05-10,2020-05-10' + . '|2020-05-11,2020-05-11|2020-05-12,2020-05-12|2020-05-13,2020-05-13|2020-05-14,2020-05-14|2020-05-15,2020-05-15' + . '|2020-05-16,2020-05-16|2020-05-17,2020-05-17|2020-05-18,2020-05-18|2020-05-19,2020-05-19|2020-05-20,2020-05-20' + . '|2020-05-21,2020-05-21|2020-05-22,2020-05-22|2020-05-23,2020-05-23|2020-05-24,2020-05-24|2020-05-25,2020-05-25' + . '|2020-05-26,2020-05-26|2020-05-27,2020-05-27|2020-05-28,2020-05-28|2020-05-29,2020-05-29|2020-05-30,2020-05-30' + . '|2020-05-31,2020-05-31|2020-06-01,2020-06-01|2020-06-02,2020-06-02|2020-06-03,2020-06-03|2020-06-04,2020-06-04' + . '|2020-06-05,2020-06-05|2020-06-06,2020-06-06|2020-06-07,2020-06-07|2020-06-08,2020-06-08|2020-06-09,2020-06-09' + . '|2020-06-10,2020-06-10|2020-06-11,2020-06-11|2020-06-12,2020-06-12|2020-06-13,2020-06-13|2020-06-14,2020-06-14', + 'count' => '45', + ), + array ( + 'idsite' => '1', + 'period' => '2', + 'name' => 'done.VisitsSummary', + 'report' => 'some.Report', + 'dates' => '2020-05-04,2020-05-10|2020-05-11,2020-05-17|2020-05-18,2020-05-24|2020-05-25,2020-05-31|2020-04-27,2020-05-03' + . '|2020-06-01,2020-06-07|2020-06-08,2020-06-14', + 'count' => '7', + ), + array ( + 'idsite' => '1', + 'period' => '3', + 'name' => 'done.VisitsSummary', + 'report' => 'some.Report', + 'dates' => '2020-05-01,2020-05-31|2020-06-01,2020-06-30', + 'count' => '2', + ), + array ( + 'idsite' => '1', + 'period' => '4', + 'name' => 'done.VisitsSummary', + 'report' => 'some.Report', + 'dates' => '2020-01-01,2020-12-31', + 'count' => '1', + ), + ]; + + $actualInvalidations = $this->getInvalidatedArchiveTableEntriesSummary(); + + $this->assertEquals($expectedInvalidations, $actualInvalidations); + } + + public function test_reArchive_alsoInvalidatesSegments() + { + Date::$now = strtotime('2020-06-16 12:00:00'); + + Config::getInstance()->General['rearchive_reports_in_past_last_n_months'] = 'last2'; + Config::getInstance()->General['process_new_segments_from'] = 'beginning_of_time'; + + $idSite = Fixture::createWebsite(Date::today()->subMonth(1)->getDatetime()); + + $t = Fixture::getTracker($idSite, '2020-05-04 03:45:45'); + $t->setUrl('http://test.com/test'); + Fixture::checkResponse($t->doTrackPageView('test page')); + + API::getInstance()->add('autoArchiveSegment', 'browserCode==IE', false, true); + API::getInstance()->add('browserArchiveSegment', 'browserCode==IE', false, false); + + $this->invalidator->reArchiveReport([$idSite], 'VisitsSummary', 'some.Report'); + + $expectedInvalidations = [ + array ( + 'idsite' => '11', + 'period' => '1', + 'name' => 'done.VisitsSummary', + 'report' => 'some.Report', + 'dates' => '2020-04-30,2020-04-30|2020-05-01,2020-05-01|2020-05-02,2020-05-02|2020-05-03,2020-05-03|2020-05-04,2020-05-04|2020-05-05,2020-05-05|2020-05-06,2020-05-06|2020-05-07,2020-05-07|2020-05-08,2020-05-08|2020-05-09,2020-05-09|2020-05-10,2020-05-10|2020-05-11,2020-05-11|2020-05-12,2020-05-12|2020-05-13,2020-05-13|2020-05-14,2020-05-14|2020-05-15,2020-05-15|2020-05-16,2020-05-16|2020-05-17,2020-05-17|2020-05-18,2020-05-18|2020-05-19,2020-05-19|2020-05-20,2020-05-20|2020-05-21,2020-05-21|2020-05-22,2020-05-22|2020-05-23,2020-05-23|2020-05-24,2020-05-24|2020-05-25,2020-05-25|2020-05-26,2020-05-26|2020-05-27,2020-05-27|2020-05-28,2020-05-28|2020-05-29,2020-05-29|2020-05-30,2020-05-30|2020-05-31,2020-05-31|2020-06-01,2020-06-01|2020-06-02,2020-06-02|2020-06-03,2020-06-03|2020-06-04,2020-06-04|2020-06-05,2020-06-05|2020-06-06,2020-06-06|2020-06-07,2020-06-07|2020-06-08,2020-06-08|2020-06-09,2020-06-09|2020-06-10,2020-06-10|2020-06-11,2020-06-11|2020-06-12,2020-06-12|2020-06-13,2020-06-13|2020-06-14,2020-06-14', + 'count' => '46', + ), + array ( + 'idsite' => '11', + 'period' => '1', + 'name' => 'done5f4f9bafeda3443c3c2d4b2ef4dffadc.VisitsSummary', + 'report' => 'some.Report', + 'dates' => '2020-04-30,2020-04-30|2020-05-01,2020-05-01|2020-05-02,2020-05-02|2020-05-03,2020-05-03|2020-05-04,2020-05-04|2020-05-05,2020-05-05|2020-05-06,2020-05-06|2020-05-07,2020-05-07|2020-05-08,2020-05-08|2020-05-09,2020-05-09|2020-05-10,2020-05-10|2020-05-11,2020-05-11|2020-05-12,2020-05-12|2020-05-13,2020-05-13|2020-05-14,2020-05-14|2020-05-15,2020-05-15|2020-05-16,2020-05-16|2020-05-17,2020-05-17|2020-05-18,2020-05-18|2020-05-19,2020-05-19|2020-05-20,2020-05-20|2020-05-21,2020-05-21|2020-05-22,2020-05-22|2020-05-23,2020-05-23|2020-05-24,2020-05-24|2020-05-25,2020-05-25|2020-05-26,2020-05-26|2020-05-27,2020-05-27|2020-05-28,2020-05-28|2020-05-29,2020-05-29|2020-05-30,2020-05-30|2020-05-31,2020-05-31|2020-06-01,2020-06-01|2020-06-02,2020-06-02|2020-06-03,2020-06-03|2020-06-04,2020-06-04|2020-06-05,2020-06-05|2020-06-06,2020-06-06|2020-06-07,2020-06-07|2020-06-08,2020-06-08|2020-06-09,2020-06-09|2020-06-10,2020-06-10|2020-06-11,2020-06-11|2020-06-12,2020-06-12|2020-06-13,2020-06-13|2020-06-14,2020-06-14', + 'count' => '46', + ), + array ( + 'idsite' => '11', + 'period' => '2', + 'name' => 'done.VisitsSummary', + 'report' => 'some.Report', + 'dates' => '2020-04-27,2020-05-03|2020-05-04,2020-05-10|2020-05-11,2020-05-17|2020-05-18,2020-05-24|2020-05-25,2020-05-31|2020-06-01,2020-06-07|2020-06-08,2020-06-14', + 'count' => '7', + ), + array ( + 'idsite' => '11', + 'period' => '2', + 'name' => 'done5f4f9bafeda3443c3c2d4b2ef4dffadc.VisitsSummary', + 'report' => 'some.Report', + 'dates' => '2020-04-27,2020-05-03|2020-05-04,2020-05-10|2020-05-11,2020-05-17|2020-05-18,2020-05-24|2020-05-25,2020-05-31|2020-06-01,2020-06-07|2020-06-08,2020-06-14', + 'count' => '7', + ), + array ( + 'idsite' => '11', + 'period' => '3', + 'name' => 'done.VisitsSummary', + 'report' => 'some.Report', + 'dates' => '2020-04-01,2020-04-30|2020-05-01,2020-05-31|2020-06-01,2020-06-30', + 'count' => '3', + ), + array ( + 'idsite' => '11', + 'period' => '3', + 'name' => 'done5f4f9bafeda3443c3c2d4b2ef4dffadc.VisitsSummary', + 'report' => 'some.Report', + 'dates' => '2020-04-01,2020-04-30|2020-05-01,2020-05-31|2020-06-01,2020-06-30', + 'count' => '3', + ), + array ( + 'idsite' => '11', + 'period' => '4', + 'name' => 'done.VisitsSummary', + 'report' => 'some.Report', + 'dates' => '2020-01-01,2020-12-31', + 'count' => '1', + ), + array ( + 'idsite' => '11', + 'period' => '4', + 'name' => 'done5f4f9bafeda3443c3c2d4b2ef4dffadc.VisitsSummary', + 'report' => 'some.Report', + 'dates' => '2020-01-01,2020-12-31', + 'count' => '1', + ), + ]; + + $actualInvalidations = $this->getInvalidatedArchiveTableEntriesSummary(); + + $this->assertEquals($expectedInvalidations, $actualInvalidations); + } + private function getInvalidatedIdArchives() { $result = array(); @@ -978,9 +1250,9 @@ class ArchiveInvalidatorTest extends IntegrationTestCase } $rangePeriods = array( - '2015-03-04,2015-03-05', - '2014-12-05,2015-01-01', - '2015-03-05,2015-03-10', + '2015-03-04,2015-03-05', + '2014-12-05,2015-01-01', + '2015-03-05,2015-03-10', '2015-01-01,2015-01-10', '2014-10-15,2014-10-20' ); @@ -1021,7 +1293,7 @@ class ArchiveInvalidatorTest extends IntegrationTestCase private function getInvalidatedArchiveTableEntries() { - return Db::fetchAll("SELECT idarchive, idsite, date1, date2, period, name FROM " . Common::prefixTable('archive_invalidations')); + return Db::fetchAll("SELECT idarchive, idsite, date1, date2, period, name, report FROM " . Common::prefixTable('archive_invalidations')); } private function assertEqualsSorted(array $expectedEntries, array $invalidatedArchiveTableEntries) @@ -1038,4 +1310,28 @@ class ArchiveInvalidatorTest extends IntegrationTestCase return strcmp(json_encode($lhs), json_encode($rhs)); }); } + + private function getInvalidatedArchiveTableEntriesSummary() + { + $table = Common::prefixTable('archive_invalidations'); + return Db::fetchAll("SELECT idsite, period, name, report, GROUP_CONCAT(CONCAT(date1, ',', date2) SEPARATOR '|') as dates, COUNT(*) as count FROM $table GROUP BY idsite, period, name, report"); + } + + private static function addVisitToEachSite() + { + $t = Fixture::getTracker(1, '2012-04-05 00:00:00'); + $t->enableBulkTracking(); + for ($i = 0; $i < 10; ++$i) { + $t->setIdSite($i + 1); + $t->setUrl('http://test.com'); + self::assertTrue($t->doTrackPageView('test page')); + } + Fixture::checkBulkTrackingResponse($t->doBulkTrack()); + } + + protected static function configureFixture($fixture) + { + parent::configureFixture($fixture); + $fixture->createSuperUser = true; + } } diff --git a/tests/PHPUnit/Integration/DataAccess/ArchiveSelectorTest.php b/tests/PHPUnit/Integration/DataAccess/ArchiveSelectorTest.php index 408ba7910a..6eaf1d9a8a 100644 --- a/tests/PHPUnit/Integration/DataAccess/ArchiveSelectorTest.php +++ b/tests/PHPUnit/Integration/DataAccess/ArchiveSelectorTest.php @@ -52,8 +52,13 @@ class ArchiveSelectorTest extends IntegrationTestCase private function insertArchiveData($archiveRows) { - $table = ArchiveTableCreator::getNumericTable(Date::factory('2019-10-01 12:13:14')); foreach ($archiveRows as $row) { + if (!empty($row['is_blob_data'])) { + $row['value'] = gzcompress($row['value']); + } + + $d = Date::factory($row['date1']); + $table = !empty($row['is_blob_data']) ? ArchiveTableCreator::getBlobTable($d) : ArchiveTableCreator::getNumericTable($d); $tsArchived = isset($row['ts_archived']) ? $row['ts_archived'] : Date::now()->getDatetime(); Db::query("INSERT INTO `$table` (idarchive, idsite, period, date1, date2, `name`, `value`, ts_archived) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", [$row['idarchive'], $row['idsite'], $row['period'], $row['date1'], $row['date2'], $row['name'], $row['value'], $tsArchived]); @@ -143,7 +148,7 @@ class ArchiveSelectorTest extends IntegrationTestCase '', $minDateProcessed, false, - [false, 0, 0, true], + [false, false, false, true], ], // archive is too old @@ -201,7 +206,7 @@ class ArchiveSelectorTest extends IntegrationTestCase '', $minDateProcessed, false, - [1, false, false, true], + [[1], 0, 0, true], ], [ [ @@ -212,7 +217,218 @@ class ArchiveSelectorTest extends IntegrationTestCase '', $minDateProcessed, false, - [1, 5, 10, true], + [[1], 5, 10, true], + ], + ]; + } + + /** + * @dataProvider getTestDataForGetArchiveData + */ + public function test_getArchiveData_returnsCorrectData($archiveRows, $dataType, $idArchives, $recordNames, $idSubtable, + $expectedData) + { + Fixture::createWebsite('2010-02-02 00:00:00'); + + $this->insertArchiveData($archiveRows); + + $data = ArchiveSelector::getArchiveData($idArchives, $recordNames, $dataType, $idSubtable); + + $this->assertEquals($expectedData, $data); + } + + public function getTestDataForGetArchiveData() + { + // $blobArray1 + $blobArray1 = [ + 1 => 'blobvalue1', + 2 => 'blobvalue2', + 3 => 'blobvalue3', + ]; + $blobArray2 = [ + 1 => 'blobvalue4', + 2 => 'blobvalue5', + 3 => 'blobvalue6', + ]; + $blobArray3 = [ + 1 => 'blobvalue7', + 2 => 'blobvalue8', + 4 => 'blobvalue9', + ]; + $blobArray4 = [ + 1 => 'blobvalue10', + 2 => 'blobvalue11', + 3 => 'blobvalue12', + ]; + + return [ + // numeric data + [ + [ + ['idarchive' => 1, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'nb_visits', 'value' => 5, 'ts_archived' => '2020-06-13 09:04:56'], + ['idarchive' => 1, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'nb_visits_converted', 'value' => 10, 'ts_archived' => '2020-06-12 02:04:56'], + ['idarchive' => 2, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'nb_visits', 'value' => 15, 'ts_archived' => '2020-06-13 04:04:56'], + ['idarchive' => 2, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'nb_visits_converted', 'value' => 20, 'ts_archived' => '2020-06-13 04:04:56'], + ['idarchive' => 3, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'nb_visits', 'value' => 30, 'ts_archived' => '2020-06-13 04:04:56'], + ], + 'numeric', + [ + '2019-10-05,2019-10-05' => [1,2,3], + ], + ['nb_visits', 'nb_visits_converted'], + null, + array ( + array ( + 'value' => '10', + 'name' => 'nb_visits_converted', + 'idsite' => '1', + 'date1' => '2019-10-05', + 'date2' => '2019-10-05', + 'ts_archived' => '2020-06-12 02:04:56', + ), + array ( + 'value' => '15', + 'name' => 'nb_visits', + 'idsite' => '1', + 'date1' => '2019-10-05', + 'date2' => '2019-10-05', + 'ts_archived' => '2020-06-13 04:04:56', + ), + array ( + 'value' => '20', + 'name' => 'nb_visits_converted', + 'idsite' => '1', + 'date1' => '2019-10-05', + 'date2' => '2019-10-05', + 'ts_archived' => '2020-06-13 04:04:56', + ), + array ( + 'value' => '30', + 'name' => 'nb_visits', + 'idsite' => '1', + 'date1' => '2019-10-05', + 'date2' => '2019-10-05', + 'ts_archived' => '2020-06-13 04:04:56', + ), + array ( + 'value' => '5', + 'name' => 'nb_visits', + 'idsite' => '1', + 'date1' => '2019-10-05', + 'date2' => '2019-10-05', + 'ts_archived' => '2020-06-13 09:04:56', + ), + ), + ], + + // blob data + [ + [ + ['idarchive' => 1, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'blob1', 'value' => 'nop', 'ts_archived' => '2020-06-13 09:04:56', 'is_blob_data' => true], + ['idarchive' => 1, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'blob2', 'value' => 'klm', 'ts_archived' => '2020-06-12 02:04:56', 'is_blob_data' => true], + ['idarchive' => 2, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'blob1', 'value' => 'hij', 'ts_archived' => '2020-06-13 04:04:56', 'is_blob_data' => true], + ['idarchive' => 2, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'blob2', 'value' => 'ghi', 'ts_archived' => '2020-06-13 04:04:56', 'is_blob_data' => true], + ['idarchive' => 3, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'blob2', 'value' => 'abcd', 'ts_archived' => '2020-06-13 04:04:56', 'is_blob_data' => true], + ['idarchive' => 4, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'blob2', 'value' => 'abcd', 'ts_archived' => '2020-08-13 04:04:56', 'is_blob_data' => true], + ], + 'blob', + [ + '2019-10-05,2019-10-05' => [1,2,3], + ], + ['blob1', 'blob2'], + null, + array ( + array ( + 'value' => 'klm', + 'name' => 'blob2', + 'idsite' => '1', + 'date1' => '2019-10-05', + 'date2' => '2019-10-05', + 'ts_archived' => '2020-06-12 02:04:56', + ), + array ( + 'value' => 'hij', + 'name' => 'blob1', + 'idsite' => '1', + 'date1' => '2019-10-05', + 'date2' => '2019-10-05', + 'ts_archived' => '2020-06-13 04:04:56', + ), + array ( + 'value' => 'ghi', + 'name' => 'blob2', + 'idsite' => '1', + 'date1' => '2019-10-05', + 'date2' => '2019-10-05', + 'ts_archived' => '2020-06-13 04:04:56', + ), + array ( + 'value' => 'abcd', + 'name' => 'blob2', + 'idsite' => '1', + 'date1' => '2019-10-05', + 'date2' => '2019-10-05', + 'ts_archived' => '2020-06-13 04:04:56', + ), + array ( + 'value' => 'nop', + 'name' => 'blob1', + 'idsite' => '1', + 'date1' => '2019-10-05', + 'date2' => '2019-10-05', + 'ts_archived' => '2020-06-13 09:04:56', + ), + ), + ], + + // blub data w/ subtable + [ + [ + ['idarchive' => 1, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'blob1_chunk_0_99', 'value' => serialize($blobArray1), 'ts_archived' => '2020-06-13 09:04:56', 'is_blob_data' => true], + ['idarchive' => 1, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'blob2_chunk_0_99', 'value' => serialize($blobArray2), 'ts_archived' => '2020-06-12 02:04:56', 'is_blob_data' => true], + ['idarchive' => 2, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'blob1_chunk_0_99', 'value' => serialize($blobArray3), 'ts_archived' => '2020-06-13 04:04:56', 'is_blob_data' => true], + ['idarchive' => 2, 'idsite' => 1, 'period' => 1, 'date1' => '2019-10-05', 'date2' => '2019-10-05', 'name' => 'blob2_chunk_0_99', 'value' => serialize($blobArray4), 'ts_archived' => '2020-06-13 04:04:56', 'is_blob_data' => true], + ], + 'blob', + [ + '2019-10-05,2019-10-05' => [1,2,3], + ], + ['blob1', 'blob2'], + 2, + array ( + array ( + 'value' => 'blobvalue5', + 'name' => 'blob2_2', + 'idsite' => '1', + 'date1' => '2019-10-05', + 'date2' => '2019-10-05', + 'ts_archived' => '2020-06-12 02:04:56', + ), + array ( + 'value' => 'blobvalue8', + 'name' => 'blob1_2', + 'idsite' => '1', + 'date1' => '2019-10-05', + 'date2' => '2019-10-05', + 'ts_archived' => '2020-06-13 04:04:56', + ), + array ( + 'value' => 'blobvalue11', + 'name' => 'blob2_2', + 'idsite' => '1', + 'date1' => '2019-10-05', + 'date2' => '2019-10-05', + 'ts_archived' => '2020-06-13 04:04:56', + ), + array ( + 'value' => 'blobvalue2', + 'name' => 'blob1_2', + 'idsite' => '1', + 'date1' => '2019-10-05', + 'date2' => '2019-10-05', + 'ts_archived' => '2020-06-13 09:04:56', + ), + ), ], ]; } diff --git a/tests/PHPUnit/Integration/DataAccess/ArchiveWriterTest.php b/tests/PHPUnit/Integration/DataAccess/ArchiveWriterTest.php index d227e6f91e..d8493d14ad 100644 --- a/tests/PHPUnit/Integration/DataAccess/ArchiveWriterTest.php +++ b/tests/PHPUnit/Integration/DataAccess/ArchiveWriterTest.php @@ -18,9 +18,10 @@ use Piwik\Db; use Piwik\Period\Day; use Piwik\Period\Factory as PeriodFactory; use Piwik\Segment; +use Piwik\Sequence; use Piwik\Site; use Piwik\Tests\Framework\Fixture; -use Piwik\Tests\Integration\Settings\IntegrationTestCase; +use Piwik\Tests\Framework\TestCase\IntegrationTestCase; class TestArchiveWriter extends ArchiveWriter { public function flushSpools() @@ -40,24 +41,70 @@ class ArchiveWriterTest extends IntegrationTestCase public function setUp(): void { + parent::setUp(); + Access::getInstance()->setSuperUserAccess(true); $this->idSite = Fixture::createWebsite('2019-08-29'); } - public function test_initNewArchive_doesNotWiteNewArchiveStatusToFileRightAway() + public function test_finalizeArchive_removesOldArchivesIfNotPartial() { + Date::$now = strtotime('2020-04-05 03:00:00'); + $period = 'day'; $date = '2019-08-29'; + $initialArchiveData = [ + ['idarchive' => 1, 'idsite' => $this->idSite, 'date1' => '2019-08-29', 'date2' => '2019-08-29', 'period' => 1, 'name' => 'done', 'value' => ArchiveWriter::DONE_PARTIAL, 'ts_archived' => '2020-02-02 03:44:44'], + ['idarchive' => 2, 'idsite' => $this->idSite, 'date1' => '2019-08-29', 'date2' => '2019-08-29', 'period' => 1, 'name' => 'done', 'value' => ArchiveWriter::DONE_OK, 'ts_archived' => '2020-02-04 03:44:44'], + ]; + + $this->insertArchiveData($initialArchiveData); + $writer = $this->buildWriter($period, $date); $writer->initNewArchive(); + $writer->insertRecord('nb_visits', 5); - $this->assertEquals(array(), $this->getAllNumericRows($date)); + $this->assertEquals($initialArchiveData, $this->getAllColsOfAllNumericRows($date)); - // now we flush and it should be written - $writer->flushSpools(); - $this->assertCount(1, $this->getAllNumericRows($date)); - $this->assertNumericArchiveExists(Day::PERIOD_ID, $date, 'done', ArchiveWriter::DONE_ERROR); + $writer->finalizeArchive(); + + $expected = [ + ['idarchive' => 3, 'idsite' => 1, 'date1' => '2019-08-29', 'date2' => '2019-08-29', 'period' => 1, 'name' => 'done', 'value' => 1, 'ts_archived' => '2020-04-05 03:00:00'], + ['idarchive' => 3, 'idsite' => 1, 'date1' => '2019-08-29', 'date2' => '2019-08-29', 'period' => 1, 'name' => 'nb_visits', 'value' => 5, 'ts_archived' => '2020-04-05 03:00:00'], + ]; + $this->assertEquals($expected, $this->getAllColsOfAllNumericRows($date)); + } + + public function test_finalizeArchive_doesNotRemoveOldArchivesIfPartial() + { + Date::$now = strtotime('2020-04-05 03:00:00'); + + $period = 'day'; + $date = '2019-08-29'; + + $initialArchiveData = [ + ['idarchive' => 1, 'idsite' => $this->idSite, 'date1' => '2019-08-29', 'date2' => '2019-08-29', 'period' => 1, 'name' => 'done', 'value' => ArchiveWriter::DONE_OK, 'ts_archived' => '2020-02-02 03:44:44'], + ['idarchive' => 2, 'idsite' => $this->idSite, 'date1' => '2019-08-29', 'date2' => '2019-08-29', 'period' => 1, 'name' => 'done', 'value' => ArchiveWriter::DONE_PARTIAL, 'ts_archived' => '2020-02-04 03:44:44'], + ]; + + $this->insertArchiveData($initialArchiveData); + + $writer = $this->buildWriter($period, $date, $isPartial = true); + $writer->initNewArchive(); + $writer->insertRecord('nb_visits', 5); + + $this->assertEquals($initialArchiveData, $this->getAllColsOfAllNumericRows($date)); + + $writer->finalizeArchive(); + + $expected = [ + ['idarchive' => 1, 'idsite' => $this->idSite, 'date1' => '2019-08-29', 'date2' => '2019-08-29', 'period' => 1, 'name' => 'done', 'value' => ArchiveWriter::DONE_OK, 'ts_archived' => '2020-02-02 03:44:44'], + ['idarchive' => 2, 'idsite' => $this->idSite, 'date1' => '2019-08-29', 'date2' => '2019-08-29', 'period' => 1, 'name' => 'done', 'value' => ArchiveWriter::DONE_PARTIAL, 'ts_archived' => '2020-02-04 03:44:44'], + ['idarchive' => 3, 'idsite' => 1, 'date1' => '2019-08-29', 'date2' => '2019-08-29', 'period' => 1, 'name' => 'done', 'value' => 5, 'ts_archived' => '2020-04-05 03:00:00'], + ['idarchive' => 3, 'idsite' => 1, 'date1' => '2019-08-29', 'date2' => '2019-08-29', 'period' => 1, 'name' => 'nb_visits', 'value' => 5, 'ts_archived' => '2020-04-05 03:00:00'], + ]; + $this->assertEquals($expected, $this->getAllColsOfAllNumericRows($date)); } public function test_finaliseArchive_writesArchiveStatusToFile() @@ -72,6 +119,22 @@ class ArchiveWriterTest extends IntegrationTestCase $this->assertNumericArchiveExists(Day::PERIOD_ID, $date, 'done', ArchiveWriter::DONE_OK); } + public function test_initNewArchive_doesNotWiteNewArchiveStatusToFileRightAway() + { + $period = 'day'; + $date = '2019-08-29'; + + $writer = $this->buildWriter($period, $date); + $writer->initNewArchive(); + + $this->assertEquals(array(), $this->getAllNumericRows($date)); + + // now we flush and it should be written + $writer->flushSpools(); + $this->assertCount(1, $this->getAllNumericRows($date)); + $this->assertNumericArchiveExists(Day::PERIOD_ID, $date, 'done', ArchiveWriter::DONE_ERROR); + } + public function test_insertRecord_notFlushedUntilFinaliseCalled() { $period = 'day'; @@ -163,11 +226,15 @@ class ArchiveWriterTest extends IntegrationTestCase } } - private function buildWriter($period, $date) + private function buildWriter($period, $date, $isPartial = false) { $oPeriod = PeriodFactory::makePeriodFromQueryParams('UTC', $period, $date); $segment = new Segment('', []); $params = new Parameters(new Site($this->idSite), $oPeriod, $segment); + if ($isPartial) { + $params->setRequestedPlugin('ExamplePlugin'); + $params->setIsPartialArchive(true); + } $writer = new TestArchiveWriter($params); return $writer; } @@ -218,4 +285,34 @@ class ArchiveWriterTest extends IntegrationTestCase $result = Db::get()->query($sql); return $result->fetch(); } + + private function insertArchiveData($archiveRows) + { + if (!empty($archiveRows)) { + $idarchives = array_column($archiveRows, 'idarchive'); + $max = max($idarchives); + + $d = Date::factory($archiveRows[0]['date1']); + $tableName = Common::prefixTable('archive_numeric_' . $d->toString('Y_m')); + $seq = new Sequence($tableName); + $seq->create($max); + } + + foreach ($archiveRows as $row) { + $d = Date::factory($row['date1']); + $table = ArchiveTableCreator::getNumericTable($d); + $tsArchived = isset($row['ts_archived']) ? $row['ts_archived'] : Date::now()->getDatetime(); + + Db::query("INSERT INTO `$table` (idarchive, idsite, period, date1, date2, `name`, `value`, ts_archived) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", + [$row['idarchive'], $row['idsite'], $row['period'], $row['date1'], $row['date2'], $row['name'], $row['value'], $tsArchived]); + } + } + + private function getAllColsOfAllNumericRows(string $date) + { + $archiveTableName = ArchiveTableCreator::getNumericTable(Date::factory($date)); + $sql = 'SELECT idarchive, idsite, date1, date2, period, name, value, ts_archived FROM ' . $archiveTableName; + + return Db::fetchAll($sql); + } } diff --git a/tests/PHPUnit/Integration/DataAccess/ModelTest.php b/tests/PHPUnit/Integration/DataAccess/ModelTest.php index b2377bbf8d..42f67fea92 100644 --- a/tests/PHPUnit/Integration/DataAccess/ModelTest.php +++ b/tests/PHPUnit/Integration/DataAccess/ModelTest.php @@ -263,6 +263,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-04-24', 'period' => '5', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '12', @@ -272,6 +273,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-04-06', 'period' => '1', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '13', @@ -281,6 +283,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-04-06', 'period' => '1', 'name' => 'done764644a7142bdcbedaab92f9dedef5e5', + 'report' => null, ), array ( 'idinvalidation' => '19', @@ -290,6 +293,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-04-12', 'period' => '2', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '5', @@ -299,6 +303,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-04-12', 'period' => '2', 'name' => 'done764644a7142bdcbedaab92f9dedef5e5', + 'report' => null, ), array ( 'idinvalidation' => '15', @@ -308,6 +313,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-04-05', 'period' => '1', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '8', @@ -317,6 +323,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-04-04', 'period' => '1', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '14', @@ -326,6 +333,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-04-03', 'period' => '1', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '20', @@ -335,6 +343,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-04-02', 'period' => '1', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '3', @@ -344,6 +353,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-04-02', 'period' => '1', 'name' => 'done67564f109e3f4bba6b185a5343ff2bb0', + 'report' => null, ), array ( 'idinvalidation' => '2', @@ -353,6 +363,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-04-01', 'period' => '1', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '10', @@ -362,6 +373,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-04-30', 'period' => '3', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '17', @@ -371,6 +383,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-04-30', 'period' => '3', 'name' => 'done67564f109e3f4bba6b185a5343ff2bb0', + 'report' => null, ), array ( 'idinvalidation' => '22', @@ -380,6 +393,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-03-31', 'period' => '1', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '7', @@ -389,6 +403,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-03-30', 'period' => '1', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '1', @@ -398,6 +413,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-03-30', 'period' => '1', 'name' => 'done0bb102ea2ac682a578480dd184736607', + 'report' => null, ), array ( 'idinvalidation' => '16', @@ -407,6 +423,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-04-05', 'period' => '2', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '6', @@ -416,6 +433,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-03-29', 'period' => '1', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '9', @@ -425,6 +443,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-03-29', 'period' => '1', 'name' => 'doneb321434abb5a139c17dadf08c9d2e315', + 'report' => null, ), array ( 'idinvalidation' => '18', @@ -434,6 +453,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-03-24', 'period' => '1', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '21', @@ -443,6 +463,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-03-31', 'period' => '3', 'name' => 'done', + 'report' => null, ), array ( 'idinvalidation' => '4', @@ -452,6 +473,7 @@ class ModelTest extends IntegrationTestCase 'date2' => '2015-12-31', 'period' => '4', 'name' => 'done', + 'report' => null, ), ); diff --git a/tests/PHPUnit/System/OneVisitorOneWebsiteSeveralDaysDateRangeArchivingTest.php b/tests/PHPUnit/System/OneVisitorOneWebsiteSeveralDaysDateRangeArchivingTest.php index efce6d829c..f5be1dd491 100644 --- a/tests/PHPUnit/System/OneVisitorOneWebsiteSeveralDaysDateRangeArchivingTest.php +++ b/tests/PHPUnit/System/OneVisitorOneWebsiteSeveralDaysDateRangeArchivingTest.php @@ -121,9 +121,9 @@ class OneVisitorOneWebsiteSeveralDaysDateRangeArchivingTest extends SystemTestCa $expectedActionsBlobsWhenFlattened = $expectedActionsBlobs + 1; $tests = array( - 'archive_blob_2010_12' => ( ($expectedActionsBlobs+1) /*Actions*/ + 'archive_blob_2010_12' => ( ($expectedActionsBlobs+3) /*Actions*/ + 2 /* Resolution */ - + 2 /* VisitTime */) * 3, + + 2 /* VisitTime */) * 3 + 1, /** * In Each "Period=range" Archive, we expect following non zero numeric entries: @@ -217,6 +217,12 @@ class OneVisitorOneWebsiteSeveralDaysDateRangeArchivingTest extends SystemTestCa protected function printDebugWhenTestFails($table) { $data = Db::get()->fetchAll("SELECT * FROM " . Common::prefixTable($table) . " WHERE period = " . Piwik::$idPeriods['range'] . " ORDER BY idarchive ASC"); + if (strpos($table, 'blob') !== false) { + $data = array_map(function ($r) { + unset($r['value']); + return $r; + }, $data); + } var_export($data); $idArchives = array(); diff --git a/tests/PHPUnit/System/PeriodIsRangeDateIsLastNMetadataAndNormalAPITest.php b/tests/PHPUnit/System/PeriodIsRangeDateIsLastNMetadataAndNormalAPITest.php index d805428d19..5b0f56b109 100644 --- a/tests/PHPUnit/System/PeriodIsRangeDateIsLastNMetadataAndNormalAPITest.php +++ b/tests/PHPUnit/System/PeriodIsRangeDateIsLastNMetadataAndNormalAPITest.php @@ -27,6 +27,13 @@ class PeriodIsRangeDateIsLastNMetadataAndNormalAPITest extends SystemTestCase parent::setUpBeforeClass(); } + protected function setUp(): void + { + // test results change based on time of day for some reason + Date::$now = strtotime(date('Y-m-d') . ' 20:00:00'); + parent::setUp(); + } + /** * @dataProvider getApiForTesting */ diff --git a/tests/PHPUnit/System/expected/test_ImportLogs__ExamplePlugin.getExampleArchivedMetric_month.xml b/tests/PHPUnit/System/expected/test_ImportLogs__ExamplePlugin.getExampleArchivedMetric_month.xml new file mode 100644 index 0000000000..043911003e --- /dev/null +++ b/tests/PHPUnit/System/expected/test_ImportLogs__ExamplePlugin.getExampleArchivedMetric_month.xml @@ -0,0 +1,5 @@ +<?xml version="1.0" encoding="utf-8" ?> +<result> + <ExamplePlugin_example_metric>12814</ExamplePlugin_example_metric> + <ExamplePlugin_example_metric2>520</ExamplePlugin_example_metric2> +</result>
\ No newline at end of file diff --git a/tests/PHPUnit/System/expected/test_OneVisitorTwoVisits__ExamplePlugin.getExampleArchivedMetric_day.xml b/tests/PHPUnit/System/expected/test_OneVisitorTwoVisits__ExamplePlugin.getExampleArchivedMetric_day.xml new file mode 100644 index 0000000000..7422ab593f --- /dev/null +++ b/tests/PHPUnit/System/expected/test_OneVisitorTwoVisits__ExamplePlugin.getExampleArchivedMetric_day.xml @@ -0,0 +1,5 @@ +<?xml version="1.0" encoding="utf-8" ?> +<result> + <ExamplePlugin_example_metric>2316</ExamplePlugin_example_metric> + <ExamplePlugin_example_metric2>55</ExamplePlugin_example_metric2> +</result>
\ No newline at end of file diff --git a/tests/PHPUnit/System/expected/test_noVisit_PeriodIsLast__ExamplePlugin.getExampleArchivedMetric_day.xml b/tests/PHPUnit/System/expected/test_noVisit_PeriodIsLast__ExamplePlugin.getExampleArchivedMetric_day.xml new file mode 100644 index 0000000000..106f23f16b --- /dev/null +++ b/tests/PHPUnit/System/expected/test_noVisit_PeriodIsLast__ExamplePlugin.getExampleArchivedMetric_day.xml @@ -0,0 +1,10 @@ +<?xml version="1.0" encoding="utf-8" ?> +<results> + <result date="2009-01-04" /> + <result date="2009-01-05" /> + <result date="2009-01-06" /> + <result date="2009-01-07" /> + <result date="2009-01-08" /> + <result date="2009-01-09" /> + <result date="2009-01-10" /> +</results>
\ No newline at end of file diff --git a/tests/PHPUnit/System/expected/test_noVisit_PeriodIsLast__ExamplePlugin.getExampleArchivedMetric_week.xml b/tests/PHPUnit/System/expected/test_noVisit_PeriodIsLast__ExamplePlugin.getExampleArchivedMetric_week.xml new file mode 100644 index 0000000000..867400a47a --- /dev/null +++ b/tests/PHPUnit/System/expected/test_noVisit_PeriodIsLast__ExamplePlugin.getExampleArchivedMetric_week.xml @@ -0,0 +1,10 @@ +<?xml version="1.0" encoding="utf-8" ?> +<results> + <result date="2008-12-29,2009-01-04" /> + <result date="2009-01-05,2009-01-11" /> + <result date="2009-01-12,2009-01-18" /> + <result date="2009-01-19,2009-01-25" /> + <result date="2009-01-26,2009-02-01" /> + <result date="2009-02-02,2009-02-08" /> + <result date="2009-02-09,2009-02-15" /> +</results>
\ No newline at end of file diff --git a/tests/PHPUnit/System/expected/test_noVisit__ExamplePlugin.getExampleArchivedMetric_day.xml b/tests/PHPUnit/System/expected/test_noVisit__ExamplePlugin.getExampleArchivedMetric_day.xml new file mode 100644 index 0000000000..c82a27e763 --- /dev/null +++ b/tests/PHPUnit/System/expected/test_noVisit__ExamplePlugin.getExampleArchivedMetric_day.xml @@ -0,0 +1,5 @@ +<?xml version="1.0" encoding="utf-8" ?> +<result> + <ExamplePlugin_example_metric>0</ExamplePlugin_example_metric> + <ExamplePlugin_example_metric2>0</ExamplePlugin_example_metric2> +</result>
\ No newline at end of file diff --git a/tests/UI/expected-screenshots/UIIntegrationTest_admin_diagnostics_configfile.png b/tests/UI/expected-screenshots/UIIntegrationTest_admin_diagnostics_configfile.png index dfae4a5e04..183d50f357 100644 --- a/tests/UI/expected-screenshots/UIIntegrationTest_admin_diagnostics_configfile.png +++ b/tests/UI/expected-screenshots/UIIntegrationTest_admin_diagnostics_configfile.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ede1813eeb82cc042fc4f8244a8c2ce0210eba52b37a3c9de130e68067414b1a -size 4319181 +oid sha256:e48f0ab68a157e15ad5f27cea5c1b263ce9fb5f434e9d80f2fb190b31f2952e2 +size 4335592 diff --git a/tests/UI/expected-screenshots/UIIntegrationTest_admin_plugins.png b/tests/UI/expected-screenshots/UIIntegrationTest_admin_plugins.png index a9a0ced4ed..52afc9cd82 100644 --- a/tests/UI/expected-screenshots/UIIntegrationTest_admin_plugins.png +++ b/tests/UI/expected-screenshots/UIIntegrationTest_admin_plugins.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:27e7041385dfc642c951ad647ad056ccfcb52726e602fb05f30c7a74564e73c3 -size 1054626 +oid sha256:947f16f424732a8418f1977e8f50358100f553a100c3a46d25c2fca9c0966f41 +size 1063376 diff --git a/tests/UI/expected-screenshots/UIIntegrationTest_api_listing.png b/tests/UI/expected-screenshots/UIIntegrationTest_api_listing.png index 384bf2ed73..2449ec4c08 100644 --- a/tests/UI/expected-screenshots/UIIntegrationTest_api_listing.png +++ b/tests/UI/expected-screenshots/UIIntegrationTest_api_listing.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1a816e74a8d2033456d840d9a96c71a6ad721d900ad74c3572ae07a25d5fdab4 -size 4924205 +oid sha256:7e711ea4c2de88449879eef219e308403c05eb850e2772c96e63a21dde4d6209 +size 4936276 |
