allow invalidating plugin archives only and archiving past data for plugins (#15889)

* Adding initial new code for cron archive rewrite.

* first pass at removing unused CronArchive code.

* unfinished commit

* fill out archiveinvalidator code

* getting some tests to pass

* unfinished commit

* fixing part of test

* Another test fix.

* another sql change

* fix broken merge or something else that went wrong

* Couple more fixes and extra logs.

* Fixing enough issues to get core archive command to run completely.

* Fix and log change.

* Fixed more segment/test related issues for CronArchiveTest. Includes optimization for no visits for period + segment process from handling.

* another optimization and possible build fix

* no visit optimization

* test fix

* Implement archiving_custom_ranges logic w/ queue based implementation

* fixes to get archivecrontest to work

* add logic to invalidate today period

* fix optimization and some tests

* Fixing more tests.

* Fixing more tests

* debug travis failure

* more test fixes

* more test fixes, removing more unneeded code, handling some TODOs

* Handle more TODOs including creating ArchiveFilter class for some cli options.

* tests and todos

* idarchives are specific to table + start on archivefilter tests

* one test

* more TODOs and tests

* more tests and todo taken care of

* handle more todos

* fixing more tests

* fix comment

* make sure autoarchiving is enabled for segments when cron archive picks them up

* Fixing test.

* apply more pr feedback

* order by date1 asc

* quick refactor

* use batch insert instead of createDummyArchives

* apply rest of pr feedback

* add removed events, add new test, fix an issue (when deleting idarchives older than do not lump all segments together).

* re-add fixed/shared siteids

* fix tests

* incomplete commit

* Insert archive entries into archive_invalidations table.

* Use invalidations table in core:archive and get ArchiveCronTest to pass.

* fixing some tests

* debugging travis

* fix more tests & remove DONE_IN_PROGRESS which is no longer used.

* fix more tests

* Allow forcing plugin specific archive in core:archive.

* When querying from archive data use all available archives including "all" archives and plugin specific archives.

* Adding some code for invalidating specific plugin archives.

* Get archive invalidation test to pass.

* add plugin capability to invalidate command

* Handle plugin only archives in core:archive.

* Add Archive test and get ArchiveCronTest to pass.

* update some expected files

* Fix some more tests.

* incomplete commit

* allow invalidating individual reports

* adding more API for DONE_PARTIAL support

* get archivecrontest to pass

* add archive processor tests

* fix some test randomnes

* when purging keep latest partial archives if there is no newer whole archive

* add rearchivereport method + some unfinished tests

* Add archiveReports API method, fix race condition in test, when archiving single report, always ignore inserting other reports.

* require archivers to handle partial archives themselves entirely instead of trying to do it automatically and allow requested report to be any string

* couple fixes

* Use core config option for last N montsh to invalidate.

* Add test for ArchiveSelector method.

* Ignore archives w/ deactivated plugins.

* Refactor queue looping into new QueueConsumer class.

* apply more review feedback + another fix

* invalidate segments too in reArchiveReport w/ etsts

* remove DONE_IN_PROGRESS, no longer used. use new status in query and add queue consumer test.

* forgot to add file

* delete old unneeded archives when finalizing a new one.

* tweak invalidation archive description

* add plugin archiving tests and get them to pass

* fix test

* many fixes

* fix another test

* update expected test files

* fix more tests

* last test fixes hopefully

* tweak log

* In case a column already exists, do not try to add it in an AddColumns migration or the entire migration will fail and no columns will be added.

* try to fix tests again

* fix again?

* apply some review feedback + fix test

* fix test

* fix another test

* couple fixes

* Remove extra param.

* apply pr feedback

* check for usable archive before invalidating and before initiating archiving

* fixing tests

* fixing tests

* fixing tests

* fix another test issue

* fix archiveinvalidator test

* fix one test and debug another

* more debugging

* fix test

* use twig

* remove no longer needed change

* add back previous logic

* fix tracking is not working

* apply pr feedback and add tests

* fixing tests

* update submodule

* debugging random travis failure

* update test

* more debugging

* more debugging

* another attempt at debugging

* Lets try this fix

* trying to fix the build

* debug

* simpler test

* fix test

* fix test

* fix test

* fix test

* fix test failure

* update screenshots

* update screenshots

Co-authored-by: Thomas Steur <tsteur@users.noreply.github.com>

1 files changed, 0 insertions, 0 deletions

diff --git a/plugins/CustomVariables b/plugins/CustomVariables
-Subproject 1aee5523fb807e0d8a8cc5290e81133ca7dd563
+Subproject 6f2b1419fdf11f4ce845658e4d5d2634b61dcda