diff options
Diffstat (limited to 'doc/development/geo/framework.md')
| -rw-r--r-- | doc/development/geo/framework.md | 147 |
1 files changed, 135 insertions, 12 deletions
diff --git a/doc/development/geo/framework.md b/doc/development/geo/framework.md index de4d6fb869d..64c9030e3dd 100644 --- a/doc/development/geo/framework.md +++ b/doc/development/geo/framework.md @@ -90,6 +90,14 @@ module Geo def self.model ::Packages::PackageFile end + + # Change this to `true` to release replication of this model. Then remove + # this override in the next release. + # The feature flag follows the format `geo_#{replicable_name}_replication`, + # so here it would be `geo_package_file_replication` + def self.replication_enabled_by_default? + false + end end end ``` @@ -180,6 +188,11 @@ For example, to add support for files referenced by a `Widget` model with a mount_uploader :file, WidgetUploader + def local? + # Must to be implemented, Check the uploader's storage types + file_store == ObjectStorage::Store::LOCAL + end + def self.replicables_for_geo_node # Should be implemented. The idea of the method is to restrict # the set of synced items depending on synchronization settings @@ -206,10 +219,30 @@ For example, to add support for files referenced by a `Widget` model with a def carrierwave_uploader model_record.file end + + # Change this to `true` to release replication of this model. Then remove + # this override in the next release. + # The feature flag follows the format `geo_#{replicable_name}_replication`, + # so here it would be `geo_widget_replication` + def self.replication_enabled_by_default? + false + end end end ``` +1. Add this replicator class to the method `replicator_classes` in +`ee/lib/gitlab/geo.rb`: + + ```ruby + def self.replicator_classes + classes = [::Geo::PackageFileReplicator, + ::Geo::WidgetReplicator] + + classes.select(&:enabled?) + end + ``` + 1. Create `ee/spec/replicators/geo/widget_replicator_spec.rb` and perform the setup necessary to define the `model_record` variable for the shared examples. @@ -226,13 +259,15 @@ For example, to add support for files referenced by a `Widget` model with a end ``` -1. Create the `widget_registry` table so Geo secondaries can track the sync and - verification state of each Widget's file: +1. Create the `widget_registry` table, with columns ordered according to [our guidelines](../ordering_table_columns.md) so Geo secondaries can track the sync and + verification state of each Widget's file. This migration belongs in `ee/db/geo/migrate`: ```ruby # frozen_string_literal: true class CreateWidgetRegistry < ActiveRecord::Migration[6.0] + include Gitlab::Database::MigrationHelpers + DOWNTIME = false disable_ddl_transaction! @@ -244,12 +279,12 @@ For example, to add support for files referenced by a `Widget` model with a t.integer :widget_id, null: false t.integer :state, default: 0, null: false, limit: 2 t.integer :retry_count, default: 0, limit: 2 - t.text :last_sync_failure t.datetime_with_timezone :retry_at t.datetime_with_timezone :last_synced_at t.datetime_with_timezone :created_at, null: false + t.text :last_sync_failure - t.index :widget_id + t.index :widget_id, name: :index_widget_registry_on_widget_id t.index :retry_at t.index :state end @@ -286,6 +321,8 @@ For example, to add support for files referenced by a `Widget` model with a 1. Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`. +1. Add `widget_registry` to `ActiveSupport::Inflector.inflections` in `config/initializers_before_autoloader/000_inflections.rb`. + 1. Create `ee/spec/factories/geo/widget_registry.rb`: ```ruby @@ -343,8 +380,13 @@ Widgets should now be replicated by Geo! #### Verification -1. Add verification state fields to the `widgets` table so the Geo primary can - track verification state: +There are two ways to add verification related fields so that the Geo primary +can track verification state: + +##### Option 1: Add verification state fields to the existing `widgets` table itself + +1. Add a migration to add columns ordered according to [our guidelines](../ordering_table_columns.md) +for verification state to the widgets table: ```ruby # frozen_string_literal: true @@ -353,17 +395,39 @@ Widgets should now be replicated by Geo! DOWNTIME = false def change - add_column :widgets, :verification_retry_at, :datetime_with_timezone - add_column :widgets, :verified_at, :datetime_with_timezone - add_column :widgets, :verification_checksum, :binary, using: 'verification_checksum::bytea' - add_column :widgets, :verification_failure, :string - add_column :widgets, :verification_retry_count, :integer + change_table(:widgets) do |t| + t.integer :verification_retry_count, limit: 2 + t.column :verification_retry_at, :datetime_with_timezone + t.column :verified_at, :datetime_with_timezone + t.binary :verification_checksum, using: 'verification_checksum::bytea' + + # rubocop:disable Migration/AddLimitToTextColumns + t.text :verification_failure + # rubocop:enable Migration/AddLimitToTextColumns + end + end + end + ``` + +1. Adding a `text` column also [requires](../database/strings_and_the_text_data_type.md#add-a-text-column-to-an-existing-table) + setting a limit: + + ```ruby + # frozen_string_literal: true + + class AddVerificationFailureLimitToWidgets < ActiveRecord::Migration[6.0] + DOWNTIME = false + + disable_ddl_transaction! + + def change + add_text_limit :widgets, :verification_failure, 255 end end ``` 1. Add a partial index on `verification_failure` and `verification_checksum` to ensure - re-verification can be performed efficiently: + re-verification can be performed efficiently. Add a migration in `ee/db/geo/migrate/`: ```ruby # frozen_string_literal: true @@ -387,6 +451,65 @@ Widgets should now be replicated by Geo! end ``` +##### Option 2: Create a separate `widget_states` table with verification state fields + +1. Add a migration in `ee/db/geo/migrate/` to create a `widget_states` table and add a + partial index on `verification_failure` and `verification_checksum` to ensure + re-verification can be performed efficiently. Order the columns according to [our guidelines](../ordering_table_columns.md): + + ```ruby + # frozen_string_literal: true + + class CreateWidgetStates < ActiveRecord::Migration[6.0] + include Gitlab::Database::MigrationHelpers + + DOWNTIME = false + + disable_ddl_transaction! + + def up + unless table_exists?(:widget_states) + with_lock_retries do + create_table :widget_states, id: false do |t| + t.references :widget, primary_key: true, null: false, foreign_key: { on_delete: :cascade } + t.datetime_with_timezone :verification_retry_at + t.datetime_with_timezone :verified_at + t.integer :verification_retry_count, limit: 2 + t.binary :verification_checksum, using: 'verification_checksum::bytea' + t.text :verification_failure + + t.index :verification_failure, where: "(verification_failure IS NOT NULL)", name: "widgets_verification_failure_partial" + t.index :verification_checksum, where: "(verification_checksum IS NOT NULL)", name: "widgets_verification_checksum_partial" + end + end + end + + add_text_limit :widget_states, :verification_failure, 255 + end + + def down + drop_table :widget_states + end + end + ``` + +1. Add the following lines to the `widget` model: + + ```ruby + class Widget < ApplicationRecord + ... + has_one :widget_state, inverse_of: :widget + + delegate :verification_retry_at, :verification_retry_at=, + :verified_at, :verified_at=, + :verification_checksum, :verification_checksum=, + :verification_failure, :verification_failure=, + :verification_retry_count, :verification_retry_count=, + to: :widget_state + ... + end + ``` + To do: Add verification on secondaries. This should be done as part of [Geo: Self Service Framework - First Implementation for Package File verification](https://gitlab.com/groups/gitlab-org/-/epics/1817) |