Changeset View
Changeset View
Standalone View
Standalone View
data/common/common.yaml
| Show First 20 Lines • Show All 1,744 Lines • ▼ Show 20 Lines | swh::deploy::journal::writer::config: | ||||
| producer_config: | producer_config: | ||||
| message.max.bytes: 1000000000 | message.max.bytes: 1000000000 | ||||
| swh::deploy::journal::backfill::config_file: "%{hiera('swh::deploy::journal::conf_directory')}/backfill.yml" | swh::deploy::journal::backfill::config_file: "%{hiera('swh::deploy::journal::conf_directory')}/backfill.yml" | ||||
| swh::deploy::journal::backfill::user: swhstorage | swh::deploy::journal::backfill::user: swhstorage | ||||
| swh::deploy::journal::backfill::group: swhstorage | swh::deploy::journal::backfill::group: swhstorage | ||||
| swh::deploy::storage::db::config::read-only: "host=%{hiera('swh::deploy::storage::db::host')} port=%{hiera('swh::deploy::storage::db::port')} user=guest dbname=%{hiera('swh::deploy::storage::db::dbname')} password=guest" | swh::deploy::storage::db::config::read-only: "host=%{hiera('swh::deploy::storage::db::host')} port=%{hiera('swh::deploy::storage::db::port')} user=guest dbname=%{hiera('swh::deploy::storage::db::dbname')} password=guest" | ||||
| swh::deploy::storage::db::secondary::config::read-only: "host=somerset.internal.softwareheritage.org port=%{hiera('swh::deploy::storage::db::port')} user=guest dbname=%{hiera('swh::deploy::storage::db::dbname')} password=guest" | |||||
| swh::deploy::storage::config::local-read-only: | swh::deploy::storage::config::local-read-only: | ||||
| cls: postgresql | cls: postgresql | ||||
| db: "%{alias('swh::deploy::storage::db::config::read-only')}" | db: "%{alias('swh::deploy::storage::db::config::read-only')}" | ||||
| # objstorage is required but it's not used in that context | # objstorage is required but it's not used in that context | ||||
| objstorage: | objstorage: | ||||
| cls: memory | cls: memory | ||||
| swh::deploy::journal::backfill::config: | swh::deploy::journal::backfill::config: | ||||
| ▲ Show 20 Lines • Show All 135 Lines • ▼ Show 20 Lines | |||||
| swh::deploy::scrubber::db::host: db.internal.softwareheritage.org | swh::deploy::scrubber::db::host: db.internal.softwareheritage.org | ||||
| swh::deploy::scrubber::db::port: "%{alias('swh::deploy::db::pgbouncer::port')}" | swh::deploy::scrubber::db::port: "%{alias('swh::deploy::db::pgbouncer::port')}" | ||||
| swh::deploy::scrubber::sentry_swh_package: swh.scrubber | swh::deploy::scrubber::sentry_swh_package: swh.scrubber | ||||
| swh::deploy::scrubber::sentry_environment: "%{alias('swh::deploy::environment')}" | swh::deploy::scrubber::sentry_environment: "%{alias('swh::deploy::environment')}" | ||||
| swh::deploy::scrubber::sentry_dsn: "https://%{lookup('swh::deploy::scrubber::sentry_token')}@sentry.softwareheritage.org/23" | swh::deploy::scrubber::sentry_dsn: "https://%{lookup('swh::deploy::scrubber::sentry_token')}@sentry.softwareheritage.org/23" | ||||
| swh::deploy::scrubber::checker::postgres::conf_directory: "%{hiera('swh::conf_directory')}/scrubber" | swh::deploy::scrubber::checker::postgres::conf_directory: "%{hiera('swh::conf_directory')}/scrubber" | ||||
| swh::deploy::scrubber::checker::postgres::conf_file: "%{hiera('swh::deploy::scrubber::checker::postgres::conf_directory')}/postgres.yml" | |||||
| swh::deploy::scrubber::checker::postgres::user: swhworker | swh::deploy::scrubber::checker::postgres::user: swhworker | ||||
| swh::deploy::scrubber::checker::postgres::group: swhdev | swh::deploy::scrubber::checker::postgres::group: swhdev | ||||
| swh::deploy::scrubber::checker::postgres::object_types: | swh::deploy::scrubber::checker::postgres::object_types: | ||||
| - directory | - directory | ||||
| - revision | - revision | ||||
| - release | - release | ||||
| - snapshot | - snapshot | ||||
| swh::deploy::scrubber::checker::postgres::ranges: | swh::deploy::scrubber::checker::postgres::ranges: | ||||
| - 00000000000000000000:44444444444444444444 | - 00000000000000000000:44444444444444444444 | ||||
| - 44444444444444444444:88888888888888888888 | - 44444444444444444444:88888888888888888888 | ||||
| - 88888888888888888888:bbbbbbbbbbbbbbbbbbbb | - 88888888888888888888:bbbbbbbbbbbbbbbbbbbb | ||||
| - bbbbbbbbbbbbbbbbbbbb:ffffffffffffffffffff | - bbbbbbbbbbbbbbbbbbbb:ffffffffffffffffffff | ||||
olasd: The object ids were truncated, and skewed (the last range was smaller than the other ones). | |||||
| swh::deploy::scrubber::checker::postgres::config_per_db: | |||||
| primary: | |||||
| filepath: "%{hiera('swh::deploy::scrubber::checker::postgres::conf_directory')}/primary.yml" | |||||
| config: | |||||
| scrubber_db: | |||||
| cls: postgresql | |||||
| db: "%{alias('swh::deploy::scrubber::db::config')}" | |||||
| storage: | |||||
| cls: postgresql | |||||
| db: "%{alias('swh::deploy::storage::db::config::read-only')}" | |||||
| objstorage: | |||||
| cls: noop | |||||
| # password entry in private-data | # password entry in private-data | ||||
| swh::deploy::scrubber::db::config: "host=%{hiera('swh::deploy::scrubber::db::host')} port=%{hiera('swh::deploy::db::pgbouncer::port')} dbname=%{hiera('swh::deploy::scrubber::db::dbname')} user=%{hiera('swh::deploy::scrubber::db::user')} password=%{hiera('swh::deploy::scrubber::db::password')}" | swh::deploy::scrubber::db::config: "host=%{hiera('swh::deploy::scrubber::db::host')} port=%{hiera('swh::deploy::db::pgbouncer::port')} dbname=%{hiera('swh::deploy::scrubber::db::dbname')} user=%{hiera('swh::deploy::scrubber::db::user')} password=%{hiera('swh::deploy::scrubber::db::password')}" | ||||
| swh::deploy::scrubber::checker::postgres::config: | swh::deploy::scrubber::checker::postgres::config: | ||||
| scrubber_db: | scrubber_db: | ||||
| cls: postgresql | cls: postgresql | ||||
| db: "%{alias('swh::deploy::scrubber::db::config')}" | db: "%{alias('swh::deploy::scrubber::db::config')}" | ||||
| storage: | storage: | ||||
| ▲ Show 20 Lines • Show All 1,986 Lines • Show Last 20 Lines | |||||
The object ids were truncated, and skewed (the last range was smaller than the other ones).
You could also make it a list of dicts with start/end keys to avoid having to split the range key in the puppet manifest.
Finally we may want different ranges per object type, e.g. giving more parallelism to the slow(er) scrubbing of directory entries.