allow invalidating plugin archives only and archiving past data for plugins (#15889)

* Adding initial new code for cron archive rewrite.

* first pass at removing unused CronArchive code.

* unfinished commit

* fill out archiveinvalidator code

* getting some tests to pass

* unfinished commit

* fixing part of test

* Another test fix.

* another sql change

* fix broken merge or something else that went wrong

* Couple more fixes and extra logs.

* Fixing enough issues to get core archive command to run completely.

* Fix and log change.

* Fixed more segment/test related issues for CronArchiveTest. Includes optimization for no visits for period + segment process from handling.

* another optimization and possible build fix

* no visit optimization

* test fix

* Implement archiving_custom_ranges logic w/ queue based implementation

* fixes to get archivecrontest to work

* add logic to invalidate today period

* fix optimization and some tests

* Fixing more tests.

* Fixing more tests

* debug travis failure

* more test fixes

* more test fixes, removing more unneeded code, handling some TODOs

* Handle more TODOs including creating ArchiveFilter class for some cli options.

* tests and todos

* idarchives are specific to table + start on archivefilter tests

* one test

* more TODOs and tests

* more tests and todo taken care of

* handle more todos

* fixing more tests

* fix comment

* make sure autoarchiving is enabled for segments when cron archive picks them up

* Fixing test.

* apply more pr feedback

* order by date1 asc

* quick refactor

* use batch insert instead of createDummyArchives

* apply rest of pr feedback

* add removed events, add new test, fix an issue (when deleting idarchives older than do not lump all segments together).

* re-add fixed/shared siteids

* fix tests

* incomplete commit

* Insert archive entries into archive_invalidations table.

* Use invalidations table in core:archive and get ArchiveCronTest to pass.

* fixing some tests

* debugging travis

* fix more tests & remove DONE_IN_PROGRESS which is no longer used.

* fix more tests

* Allow forcing plugin specific archive in core:archive.

* When querying from archive data use all available archives including "all" archives and plugin specific archives.

* Adding some code for invalidating specific plugin archives.

* Get archive invalidation test to pass.

* add plugin capability to invalidate command

* Handle plugin only archives in core:archive.

* Add Archive test and get ArchiveCronTest to pass.

* update some expected files

* Fix some more tests.

* incomplete commit

* allow invalidating individual reports

* adding more API for DONE_PARTIAL support

* get archivecrontest to pass

* add archive processor tests

* fix some test randomnes

* when purging keep latest partial archives if there is no newer whole archive

* add rearchivereport method + some unfinished tests

* Add archiveReports API method, fix race condition in test, when archiving single report, always ignore inserting other reports.

* require archivers to handle partial archives themselves entirely instead of trying to do it automatically and allow requested report to be any string

* couple fixes

* Use core config option for last N montsh to invalidate.

* Add test for ArchiveSelector method.

* Ignore archives w/ deactivated plugins.

* Refactor queue looping into new QueueConsumer class.

* apply more review feedback + another fix

* invalidate segments too in reArchiveReport w/ etsts

* remove DONE_IN_PROGRESS, no longer used. use new status in query and add queue consumer test.

* forgot to add file

* delete old unneeded archives when finalizing a new one.

* tweak invalidation archive description

* add plugin archiving tests and get them to pass

* fix test

* many fixes

* fix another test

* update expected test files

* fix more tests

* last test fixes hopefully

* tweak log

* In case a column already exists, do not try to add it in an AddColumns migration or the entire migration will fail and no columns will be added.

* try to fix tests again

* fix again?

* apply some review feedback + fix test

* fix test

* fix another test

* couple fixes

* Remove extra param.

* apply pr feedback

* check for usable archive before invalidating and before initiating archiving

* fixing tests

* fixing tests

* fixing tests

* fix another test issue

* fix archiveinvalidator test

* fix one test and debug another

* more debugging

* fix test

* use twig

* remove no longer needed change

* add back previous logic

* fix tracking is not working

* apply pr feedback and add tests

* fixing tests

* update submodule

* debugging random travis failure

* update test

* more debugging

* more debugging

* another attempt at debugging

* Lets try this fix

* trying to fix the build

* debug

* simpler test

* fix test

* fix test

* fix test

* fix test

* fix test failure

* update screenshots

* update screenshots

Co-authored-by: Thomas Steur <tsteur@users.noreply.github.com>

1 files changed, 44 insertions, 13 deletions

diff --git a/tests/PHPUnit/Fixtures/RawArchiveDataWithTempAndInvalidated.php b/tests/PHPUnit/Fixtures/RawArchiveDataWithTempAndInvalidated.php
index 36c1a0ba49..408a090d18 100644
--- a/tests/PHPUnit/Fixtures/RawArchiveDataWithTempAndInvalidated.php
+++ b/tests/PHPUnit/Fixtures/RawArchiveDataWithTempAndInvalidated.php
@@ -70,7 +70,6 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture
             'ts_archived' => '2015-02-15 08:12:13'
         ),
 
-
         // valid temporary
         array( // only valid
             'idarchive' => 5,
@@ -262,12 +261,35 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture
             'ts_archived' => '2015-02-09 14:13:14'
         ),
 
+        // old done partial (should be purged)
+        array(
+            'idarchive' => 22,
+            'idsite' => 1,
+            'name' => 'done',
+            'value' => ArchiveWriter::DONE_PARTIAL,
+            'date1' => '2015-02-10',
+            'date2' => '2015-02-10',
+            'period' => 1,
+            'ts_archived' => '2015-02-11 10:13:14'
+        ),
+
+        // new done partial (should not be purged)
+        array(
+            'idarchive' => 23,
+            'idsite' => 1,
+            'name' => 'done',
+            'value' => ArchiveWriter::DONE_PARTIAL,
+            'date1' => '2015-02-10',
+            'date2' => '2015-02-10',
+            'period' => 1,
+            'ts_archived' => '2015-02-11 16:13:14'
+        ),
     );
 
 
     private static $segmentArchiveData = array(
         array(
-            'idarchive' => 22,
+            'idarchive' => 24,
             'idsite' => 1,
             'name' => 'doneeb5d2797aedd15d819b1a20425982850',   // Raw segment = abcd1234abcd5678
             'value' => ArchiveWriter::DONE_OK,
@@ -277,7 +299,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture
             'ts_archived' => '2015-02-03 12:12:12'
         ),
         array(
-            'idarchive' => 23,
+            'idarchive' => 25,
             'idsite' => 1,
             'name' => 'doneeb5d2797aedd15d819b1a20425982850.MyPlugin',   // Raw segment = abcd1234abcd5678
             'value' => ArchiveWriter::DONE_OK,
@@ -287,7 +309,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture
             'ts_archived' => '2015-02-03 12:12:12'
         ),
         array(
-            'idarchive' => 24,
+            'idarchive' => 26,
             'idsite' => 2,
             'name' => 'doneeb5d2797aedd15d819b1a20425982850',   // Raw segment = abcd1234abcd5678
             'value' => ArchiveWriter::DONE_OK,
@@ -297,7 +319,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture
             'ts_archived' => '2015-02-03 12:12:12'
         ),
         array(
-            'idarchive' => 25,
+            'idarchive' => 27,
             'idsite' => 2,
             'name' => 'doneeb5d2797aedd15d819b1a20425982850.MyPlugin',   // Raw segment = abcd1234abcd5678
             'value' => ArchiveWriter::DONE_OK,
@@ -307,7 +329,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture
             'ts_archived' => '2015-02-03 12:12:12'
         ),
         array(
-            'idarchive' => 26,
+            'idarchive' => 28,
             'idsite' => 1,
             'name' => 'done1e39a89fcc269acc36bd4d7c742763ed',   // Raw segment = 9876fedc5432abcd
             'value' => ArchiveWriter::DONE_OK,
@@ -317,7 +339,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture
             'ts_archived' => '2015-02-03 12:12:12'
         ),
         array(
-            'idarchive' => 27,
+            'idarchive' => 29,
             'idsite' => 2,
             'name' => 'done00c6ee2e21a7548de6260cf72c4f4b5b',   // Raw segment = hash1
             'value' => ArchiveWriter::DONE_OK,
@@ -327,7 +349,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture
             'ts_archived' => '2015-02-03 12:12:12'
         ),
         array(
-            'idarchive' => 28,
+            'idarchive' => 30,
             'idsite' => 2,
             'name' => 'done58833651db311ba4bc11cb26b1900b0f',   // Raw segment = hash2
             'value' => ArchiveWriter::DONE_OK,
@@ -337,7 +359,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture
             'ts_archived' => '2015-02-03 12:12:12'
         ),
         array(
-            'idarchive' => 29,
+            'idarchive' => 31,
             'idsite' => 2,
             'name' => 'done58833651db311ba4bc11cb26b1900b0f.MyPlugin',  // Raw segment = hash2
             'value' => ArchiveWriter::DONE_OK,
@@ -347,7 +369,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture
             'ts_archived' => '2015-02-03 12:12:12'
         ),
         array(
-            'idarchive' => 30,
+            'idarchive' => 32,
             'idsite' => 2,
             'name' => 'done1a4ead8b39d17dfe89418452c9bba770',   // Raw segment = hash3
             'value' => ArchiveWriter::DONE_OK,
@@ -357,7 +379,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture
             'ts_archived' => '2015-02-03 12:12:12'
         ),
         array(
-            'idarchive' => 31,
+            'idarchive' => 33,
             'idsite' => 2,
             'name' => 'done1a4ead8b39d17dfe89418452c9bba770',   // Raw segment = hash3
             'value' => ArchiveWriter::DONE_OK,
@@ -371,7 +393,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture
     public static $dummyArchiveDataNoInvalidated = [
         // two archives w/ DONE_OK for a new site (no invalidated archives for site)
         [
-            'idarchive' => 32,
+            'idarchive' => 34,
             'idsite' => 4,
             'name' => 'done',
             'value' => ArchiveWriter::DONE_OK,
@@ -381,7 +403,7 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture
             'ts_archived' => '2015-02-27 10:12:12'
         ],
         [
-            'idarchive' => 33,
+            'idarchive' => 35,
             'idsite' => 4,
             'name' => 'done',
             'value' => ArchiveWriter::DONE_OK,
@@ -558,4 +580,13 @@ class RawArchiveDataWithTempAndInvalidated extends Fixture
         $expectedExistingArchives = array(15);
         $this->assertArchivesExist($expectedExistingArchives, $date);
     }
+
+    public function assertPartialArchivesPurged(Date $date)
+    {
+        $expectedPurgedArchives = [22];
+        $this->assertArchivesDoNotExist($expectedPurgedArchives, $date);
+
+        $expectedExistingArchives = [23];
+        $this->assertArchivesExist($expectedExistingArchives, $date);
+    }
 }
\ No newline at end of file