Changeset View
Changeset View
Standalone View
Standalone View
data/defaults.yaml
Show First 20 Lines • Show All 1,011 Lines • ▼ Show 20 Lines | |||||
# Default ports | # Default ports | ||||
swh::remote_service::storage::port: 5002 | swh::remote_service::storage::port: 5002 | ||||
swh::remote_service::objstorage::port: 5003 | swh::remote_service::objstorage::port: 5003 | ||||
swh::remote_service::webapp::port: 5004 | swh::remote_service::webapp::port: 5004 | ||||
swh::remote_service::vault::port: 5005 | swh::remote_service::vault::port: 5005 | ||||
swh::remote_service::deposit::port: 5006 | swh::remote_service::deposit::port: 5006 | ||||
swh::remote_service::indexer::port: 5007 | swh::remote_service::indexer::port: 5007 | ||||
swh::remote_service::scheduler::port: 5008 | swh::remote_service::scheduler::port: 5008 | ||||
swh::remote_service::search::port: 5010 | |||||
# Default backend services. Override in specific sites if needed. Configurations | # Default backend services. Override in specific sites if needed. Configurations | ||||
# are split between read-only (the default) and writable storages. In most cases | # are split between read-only (the default) and writable storages. In most cases | ||||
# overrides should only happen for read-only services. | # overrides should only happen for read-only services. | ||||
swh::remote_service::objstorage::config: "%{alias('swh::remote_service::objstorage::config::azure_readonly_with_fallback')}" | swh::remote_service::objstorage::config: "%{alias('swh::remote_service::objstorage::config::azure_readonly_with_fallback')}" | ||||
swh::remote_service::objstorage::config::writable: "%{alias('swh::remote_service::objstorage::config::uffizi')}" | swh::remote_service::objstorage::config::writable: "%{alias('swh::remote_service::objstorage::config::uffizi')}" | ||||
swh::remote_service::objstorage::config_as_dict: | swh::remote_service::objstorage::config_as_dict: | ||||
▲ Show 20 Lines • Show All 158 Lines • ▼ Show 20 Lines | swh::remote_service::storage::config::azure: | ||||
args: | args: | ||||
url: "http://storage0.euwest.azure.internal.softwareheritage.org:%{hiera('swh::remote_service::storage::port')}/" | url: "http://storage0.euwest.azure.internal.softwareheritage.org:%{hiera('swh::remote_service::storage::port')}/" | ||||
swh::remote_service::storage::config::localhost: | swh::remote_service::storage::config::localhost: | ||||
cls: remote | cls: remote | ||||
args: | args: | ||||
url: "http://localhost:%{hiera('swh::remote_service::storage::port')}/" | url: "http://localhost:%{hiera('swh::remote_service::storage::port')}/" | ||||
swh::remote_service::search::config::empty: {} | |||||
swh::remote_service::search::config::storage0: | |||||
cls: remote | |||||
args: | |||||
url: "http://storage0.euwest.azure.internal.softwareheritage.org:%{hiera('swh::remote_service::search::port')}/" | |||||
swh::remote_service::search::config: "%{alias('swh::remote_service::search::config::empty')}" | |||||
# Indexer backend configurations | # Indexer backend configurations | ||||
swh::remote_service::indexer::config::uffizi: | swh::remote_service::indexer::config::uffizi: | ||||
cls: remote | cls: remote | ||||
args: | args: | ||||
url: "http://uffizi.internal.softwareheritage.org:%{hiera('swh::remote_service::indexer::port')}/" | url: "http://uffizi.internal.softwareheritage.org:%{hiera('swh::remote_service::indexer::port')}/" | ||||
swh::remote_service::indexer::config::azure: | swh::remote_service::indexer::config::azure: | ||||
▲ Show 20 Lines • Show All 437 Lines • ▼ Show 20 Lines | swh_save_origin: | ||||
# OpenAIRE | # OpenAIRE | ||||
- 213.135.60.145 | - 213.135.60.145 | ||||
- 213.135.60.146 | - 213.135.60.146 | ||||
# in private data: | # in private data: | ||||
# deposit_basic_auth_swhworker_username | # deposit_basic_auth_swhworker_username | ||||
# deposit_basic_auth_swhworker_password | # deposit_basic_auth_swhworker_password | ||||
swh::deploy::webapp::config: | swh::deploy::webapp::config: | ||||
search: {} | search: "%{alias('swh::remote_service::search::config')}" | ||||
storage: "%{alias('swh::remote_service::storage::config')}" | storage: "%{alias('swh::remote_service::storage::config')}" | ||||
vault: "%{alias('swh::remote_service::vault::config::writable')}" | vault: "%{alias('swh::remote_service::vault::config::writable')}" | ||||
indexer_storage: "%{alias('swh::remote_service::indexer::config')}" | indexer_storage: "%{alias('swh::remote_service::indexer::config')}" | ||||
scheduler: "%{alias('swh::remote_service::scheduler::config::writable')}" | scheduler: "%{alias('swh::remote_service::scheduler::config::writable')}" | ||||
log_dir: "%{hiera('swh::deploy::webapp::conf::log_dir')}" | log_dir: "%{hiera('swh::deploy::webapp::conf::log_dir')}" | ||||
secret_key: "%{hiera('swh::deploy::webapp::conf::secret_key')}" | secret_key: "%{hiera('swh::deploy::webapp::conf::secret_key')}" | ||||
content_display_max_size: 1048576 | content_display_max_size: 1048576 | ||||
throttling: "%{alias('swh::deploy::webapp::config::throttling')}" | throttling: "%{alias('swh::deploy::webapp::config::throttling')}" | ||||
▲ Show 20 Lines • Show All 54 Lines • ▼ Show 20 Lines | |||||
swh::deploy::scheduler::remote::backend::server_names: | swh::deploy::scheduler::remote::backend::server_names: | ||||
- "%{::swh_hostname.internal_fqdn}" | - "%{::swh_hostname.internal_fqdn}" | ||||
- "%{::hostname}" | - "%{::hostname}" | ||||
- 127.0.0.1 | - 127.0.0.1 | ||||
- localhost | - localhost | ||||
- "::1" | - "::1" | ||||
swh::deploy::scheduler::remote::config: "%{alias('swh::deploy::scheduler::config::local')}" | swh::deploy::scheduler::remote::config: "%{alias('swh::deploy::scheduler::config::local')}" | ||||
swh::elasticsearch::nodes: | |||||
- host: esnode2.internal.softwareheritage.org | |||||
port: 9200 | |||||
- host: esnode3.internal.softwareheritage.org | |||||
port: 9200 | |||||
- host: esnode1.internal.softwareheritage.org | |||||
port: 9200 | |||||
swh::deploy::scheduler::archive::conf_dir: "%{hiera('swh::conf_directory')}/backend" | swh::deploy::scheduler::archive::conf_dir: "%{hiera('swh::conf_directory')}/backend" | ||||
swh::deploy::scheduler::archive::conf_file: "%{hiera('swh::deploy::scheduler::archive::conf_dir')}/elastic.yml" | swh::deploy::scheduler::archive::conf_file: "%{hiera('swh::deploy::scheduler::archive::conf_dir')}/elastic.yml" | ||||
swh::deploy::scheduler::archive::user: "%{hiera('swh::deploy::scheduler::user')}" | swh::deploy::scheduler::archive::user: "%{hiera('swh::deploy::scheduler::user')}" | ||||
swh::deploy::scheduler::archive::config: | swh::deploy::scheduler::archive::config: | ||||
scheduler: "%{alias('swh::remote_service::scheduler::config::writable')}" | scheduler: "%{alias('swh::remote_service::scheduler::config::writable')}" | ||||
elasticsearch: | elasticsearch: | ||||
cls: local | cls: local | ||||
args: | args: | ||||
index_name_prefix: swh-tasks | index_name_prefix: swh-tasks | ||||
storage_nodes: | storage_nodes: | ||||
- host: esnode2.internal.softwareheritage.org | "%{alias('swh::elasticsearch::nodes')}" | ||||
port: 9200 | |||||
- host: esnode3.internal.softwareheritage.org | |||||
port: 9200 | |||||
- host: esnode1.internal.softwareheritage.org | |||||
port: 9200 | |||||
client_options: | client_options: | ||||
sniff_on_start: false | sniff_on_start: false | ||||
sniff_on_connection_fail: true | sniff_on_connection_fail: true | ||||
http_compress: false | http_compress: false | ||||
sniffer_timeout: 60 | sniffer_timeout: 60 | ||||
# Main lister configuration | # Main lister configuration | ||||
swh::deploy::worker::lister::db::user: swh-lister | swh::deploy::worker::lister::db::user: swh-lister | ||||
▲ Show 20 Lines • Show All 2,446 Lines • ▼ Show 20 Lines | |||||
borg::repository_group: borg | borg::repository_group: borg | ||||
borg::base_path: /srv/borg | borg::base_path: /srv/borg | ||||
borg::repository_path: "%{lookup('borg::base_path')}/repositories" | borg::repository_path: "%{lookup('borg::base_path')}/repositories" | ||||
borg::repository_server: banco.internal.softwareheritage.org | borg::repository_server: banco.internal.softwareheritage.org | ||||
borg::encryption: repokey-blake2 | borg::encryption: repokey-blake2 | ||||
swh::deploy::search::sentry_swh_package: swh.search | |||||
swh::deploy::search::sentry_environment: "%{alias('swh::deploy::environment')}" | |||||
swh::deploy::search::conf_directory: "%{hiera('swh::conf_directory')}" | |||||
swh::deploy::search::conf_file: "%{hiera('swh::deploy::search::conf_directory')}/search.yml" | |||||
swh::deploy::search::user: swhstorage | |||||
swh::deploy::search::group: swhstorage | |||||
swh::deploy::search::config: | |||||
search: | |||||
cls: elasticsearch | |||||
args: | |||||
hosts: "%{alias('swh::elasticsearch::nodes')}" | |||||
ardumont: @vlorentz are those the correct elasticsearch nodes (esnode[1-3]) which have the data to lookup? | |||||
ardumontAuthorUnsubmitted Done Inline ActionsI don't find the origin index swh-search use there so it seems not. ardumont: I don't find the `origin` index swh-search use there so it seems not. | |||||
ardumontAuthorUnsubmitted Done Inline ActionsLooking back at the task hierarchy, i saw T2184 so that might explain it. ardumont: Looking back at the task hierarchy, i saw T2184 so that might explain it. | |||||
vlorentzUnsubmitted Not Done Inline ActionsThey are the right nodes, and the index exists. >>> import elasticsearch >>> es = elasticsearch.Elasticsearch(['http://esnode3.internal.softwareheritage.org:9200']) >>> list(es.indices.get('ori*')) ['origin'] vlorentz: They are the right nodes, and the index exists.
```
>>> import elasticsearch
>>> es =… | |||||
ardumontAuthorUnsubmitted Done Inline Actions@olasd told me the index exists btw ;) ardumont: @olasd told me the index exists btw ;)
So i need to improve on how i look up those indexes. | |||||
ardumontAuthorUnsubmitted Done Inline Actionslolsad /me rollseyes at himself /me looks back at [1] to see it's all there! [1] https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/backend_es.py ardumont: lolsad
/me rollseyes at himself
/me looks back at [1] to see it's all there!
[1] https… | |||||
swh::deploy::search::backend::listen::host: 127.0.0.1 | |||||
swh::deploy::search::backend::listen::port: "%{alias('swh::remote_service::search::port')}" | |||||
swh::deploy::search::backend::workers: 4 | |||||
swh::deploy::search::backend::reload_mercy: 3600 | |||||
swh::deploy::search::backend::http_keepalive: 5 | |||||
swh::deploy::search::backend::http_timeout: 3600 | |||||
swh::deploy::search::backend::max_requests: 10000 | |||||
swh::deploy::search::backend::max_requests_jitter: 1000 | |||||
swh::deploy::search::backend::server_names: | |||||
- "%{::swh_hostname.internal_fqdn}" | |||||
- "%{::hostname}" | |||||
- 127.0.0.1 | |||||
- localhost | |||||
- "::1" | |||||
@vlorentz are those the correct elasticsearch nodes (esnode[1-3]) which have the data to lookup?