Page MenuHomeSoftware Heritage

Deploy functional loader in staging area
ClosedPublic

Authored by ardumont on Mar 18 2020, 10:17 AM.

Details

Summary

@lewo heads up ;)

Related to T1991
Related to D2792 (needs to land though)

Test Plan

bin/octocatalog:

production node, noop as expected:

bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker01
Found host worker01.softwareheritage.org
WARN     -> Environment "add-validation-server-side" contained non-word characters, correcting name to add_validation_server_side
Cloning into '/tmp/swh-ocd.EYESStXn/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.EYESStXn/environments/staging/data/private'...
done.
*** Running octocatalog-diff on host worker01.softwareheritage.org
I, [2020-03-18T10:16:36.213181 #8326]  INFO -- : Catalogs compiled for worker01.softwareheritage.org
I, [2020-03-18T10:16:37.184845 #8326]  INFO -- : Diffs computed for worker01.softwareheritage.org
I, [2020-03-18T10:16:37.184887 #8326]  INFO -- : No differences
*** End octocatalog-diff on worker01.softwareheritage.org

staging node, as expected puppet wants to install functional loader:

bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker0.internal.staging.swh.network
Found host worker0.internal.staging.swh.network
WARN     -> Environment "add-validation-server-side" contained non-word characters, correcting name to add_validation_server_side
Cloning into '/tmp/swh-ocd.pWAPpNBQ/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.pWAPpNBQ/environments/staging/data/private'...
done.
*** Running octocatalog-diff on host worker0.internal.staging.swh.network
I, [2020-03-18T10:14:56.070734 #3443]  INFO -- : Catalogs compiled for worker0.internal.staging.swh.network
I, [2020-03-18T10:14:57.042299 #3443]  INFO -- : Diffs computed for worker0.internal.staging.swh.network
diff origin/production/worker0.internal.staging.swh.network current/worker0.internal.staging.swh.network
*******************************************
+ File[/etc/softwareheritage/loader_functional.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhworker"
      "mode": "0644"
      "owner": "swhworker"
      "content": >>>
---
storage:
  cls: pipeline
  steps:
  - cls: retry
  - cls: filter
  - cls: buffer
    min_batch_size:
      content: 1000
      content_bytes: 52428800
      directory: 1000
      revision: 1000
      release: 1000
  - cls: remote
    args:
      url: http://storage0.internal.staging.swh.network:5002/
max_content_size: 104857600
celery:
  task_broker: amqp://swhconsumer:swh::deploy::worker::task_broker::password@scheduler0.internal.staging.swh.network:5672/%2f
  task_queues:
  - swh.loader.package.functional.tasks.LoadFunctional
<<<
*******************************************
+ File[/etc/systemd/system/swh-worker@loader_functional.service.d/parameters.conf] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": ["Class[Systemd::Systemctl::Daemon_reload]"]
      "owner": "root"
      "show_diff": true
      "content": >>>
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=1
Environment=MAX_TASKS_PER_CHILD=5
Environment=LOGLEVEL=info
Environment=SWH_SENTRY_DSN=swh::deploy::loader_core::sentry_dsn
Environment=SWH_SENTRY_ENVIRONMENT=staging
Environment=SWH_MAIN_PACKAGE=swh.loader.core


PrivateTmp=true
<<<
*******************************************
+ File[/etc/systemd/system/swh-worker@loader_functional.service.d] =>
   parameters =>
      "ensure": "directory"
      "group": "root"
      "owner": "root"
      "purge": true
      "recurse": true
*******************************************
+ Profile::Swh::Deploy::Worker::Instance[loader_functional] =>
   parameters =>
      "ensure": "present"
      "instance_name": "loader_functional"
      "merge_policy": "deep"
      "private_tmp": true
      "sentry_name": "loader_core"
*******************************************
+ Service[swh-worker@loader_functional] =>
   parameters =>
      "enable": true
*******************************************
+ Systemd::Dropin_file[swh-worker@loader_functional/parameters.conf] =>
   parameters =>
      "daemon_reload": "lazy"
      "ensure": "present"
      "filename": "parameters.conf"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "unit": "swh-worker@loader_functional.service"
      "content": >>>
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=1
Environment=MAX_TASKS_PER_CHILD=5
Environment=LOGLEVEL=info
Environment=SWH_SENTRY_DSN=swh::deploy::loader_core::sentry_dsn
Environment=SWH_SENTRY_ENVIRONMENT=staging
Environment=SWH_MAIN_PACKAGE=swh.loader.core


PrivateTmp=true
<<<
*******************************************
*** End octocatalog-diff on worker0.internal.staging.swh.network

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.