Page MenuHomeSoftware Heritage
Differential D1519

swh-lister: Merge lister configuration and services into one
ClosedPublic
Actions

Authored by ardumont on May 29 2019, 2:50 PM.

Details

Reviewers
douardda
ftigeot
olasd
Group Reviewers
Reviewers
Commits
rSPSITE38342bebdef6: swh-lister: Merge lister configuration and service into one
Summary
  • Instantiate only one swh-worker@lister service
  • Associate the lister's configuration for it to discuss with the right db (swh-lister on belvedere)
  • Drops the lister_<techno>.pp manifests
  • Removes each lister's configuration
  • Configure the rocquenquourt nodes to deploy swh-worker@lister

Note:

This does not do any cleanups. This will have to be done later as extra
manual steps (after the puppet agent --test step).

Related T1772

Test Plan
# worker01 runs on rocq which has the `lister` worker to deploy
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to update_configuration worker01
Found host worker01.softwareheritage.org
Cloning into '/tmp/swh-ocd.6SyrsvvP/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.6SyrsvvP/environments/update_configuration/data/private'...
done.
*** Running octocatalog-diff on host worker01.softwareheritage.org
I, [2019-05-29T14:56:41.469951 #20238]  INFO -- : Catalogs compiled for worker01.softwareheritage.org
I, [2019-05-29T14:56:42.182587 #20238]  INFO -- : Diffs computed for worker01.softwareheritage.org
diff origin/production/worker01.softwareheritage.org current/worker01.softwareheritage.org
*******************************************
+ File[/etc/softwareheritage/lister.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhworker"
      "mode": "0644"
      "owner": "swhworker"
      "content": >>>
---
storage:
  cls: remote
  args:
    url: http://uffizi.internal.softwareheritage.org:5002/
scheduler:
  cls: remote
  args:
    url: http://saatchi.internal.softwareheritage.org:5008/
lister:
  cls: local
  args:
    db: postgresql://swh-lister:@belvedere.internal.softwareheritage.org:5434/swh-lister
celery:
  task_broker: amqp://swhconsumer:swh::deploy::worker::task_broker::password@rabbitmq:5672//
  task_modules:
  - swh.lister.debian.tasks
  - swh.lister.github.tasks
  - swh.lister.gitlab.tasks
  - swh.lister.npm.tasks
  - swh.lister.pypi.tasks
  task_queues:
  - swh.lister.debian.tasks.DebianListerTask
  - swh.lister.github.tasks.IncrementalGitHubLister
  - swh.lister.github.tasks.RangeGitHubLister
  - swh.lister.github.tasks.FullGitHubRelister
  - swh.lister.gitlab.tasks.IncrementalGitLabLister
  - swh.lister.gitlab.tasks.RangeGitLabLister
  - swh.lister.gitlab.tasks.FullGitLabRelister
  - swh.lister.npm.tasks.NpmListerTask
  - swh.lister.pypi.tasks.PyPIListerTask
credentials: ''
<<<
*******************************************
- File[/etc/softwareheritage/lister]
*******************************************
- File[/etc/softwareheritage/lister_debian.yml]
*******************************************
- File[/etc/softwareheritage/lister_github.yml]
*******************************************
- File[/etc/softwareheritage/lister_gitlab.yml]
*******************************************
- File[/etc/softwareheritage/lister_npm.yml]
*******************************************
- File[/etc/softwareheritage/lister_pypi.yml]
*******************************************
  File[/etc/softwareheritage/loader_debian.yml] =>
   parameters =>
     content =>
      @@ -5,5 +5,5 @@
           url: http://uffizi.internal.softwareheritage.org:5002/
       directory_packet_size: 100
      -lister_db_url: postgresql://lister-debian:swh::deploy::loader_debian::db::password@belvedere.internal.softwareheritage.org:5434/lister-debian
      +lister_db_url: postgresql://swh-lister:swh::deploy::loader_debian::db::password@belvedere.internal.softwareheritage.org:5434/swh-lister
       celery:
         task_broker: amqp://swhconsumer:swh::deploy::worker::task_broker::password@rabbitmq:5672//
*******************************************
+ File[/etc/systemd/system/swh-worker@lister.service.d/parameters.conf] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": "Class[Systemd::Systemctl::Daemon_reload]"
      "owner": "root"
      "content": >>>
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=5
Environment=MAX_TASKS_PER_CHILD=5
Environment=LOGLEVEL=warning
<<<
*******************************************
+ File[/etc/systemd/system/swh-worker@lister.service.d] =>
   parameters =>
      "ensure": "directory"
      "group": "root"
      "owner": "root"
*******************************************
- File[/etc/systemd/system/swh-worker@lister_debian.service.d/parameters.conf]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_debian.service.d]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_github.service.d/parameters.conf]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_github.service.d]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_gitlab.service.d/parameters.conf]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_gitlab.service.d]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_npm.service.d/parameters.conf]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_npm.service.d]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_pypi.service.d/parameters.conf]
*******************************************
- File[/etc/systemd/system/swh-worker@lister_pypi.service.d]
*******************************************
+ Profile::Swh::Deploy::Worker::Instance[lister] =>
   parameters =>
      "ensure": "present"
      "instance_name": "lister"
      "max_tasks_per_child": 5
*******************************************
- Profile::Swh::Deploy::Worker::Instance[lister_debian]
*******************************************
- Profile::Swh::Deploy::Worker::Instance[lister_github]
*******************************************
- Profile::Swh::Deploy::Worker::Instance[lister_gitlab]
*******************************************
- Profile::Swh::Deploy::Worker::Instance[lister_npm]
*******************************************
- Profile::Swh::Deploy::Worker::Instance[lister_pypi]
*******************************************
+ Service[swh-worker@lister] =>
   parameters =>
      "enable": true
*******************************************
- Service[swh-worker@lister_debian]
*******************************************
- Service[swh-worker@lister_github]
*******************************************
- Service[swh-worker@lister_gitlab]
*******************************************
- Service[swh-worker@lister_npm]
*******************************************
- Service[swh-worker@lister_pypi]
*******************************************
+ Systemd::Dropin_file[swh-worker@lister/parameters.conf] =>
   parameters =>
      "ensure": "present"
      "filename": "parameters.conf"
      "path": "/etc/systemd/system"
      "unit": "swh-worker@lister.service"
      "content": >>>
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=5
Environment=MAX_TASKS_PER_CHILD=5
Environment=LOGLEVEL=warning
<<<
*******************************************
- Systemd::Dropin_file[swh-worker@lister_debian/parameters.conf]
*******************************************
- Systemd::Dropin_file[swh-worker@lister_github/parameters.conf]
*******************************************
- Systemd::Dropin_file[swh-worker@lister_gitlab/parameters.conf]
*******************************************
- Systemd::Dropin_file[swh-worker@lister_npm/parameters.conf]
*******************************************
- Systemd::Dropin_file[swh-worker@lister_pypi/parameters.conf]
*******************************************
*** End octocatalog-diff on worker01.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Branch
update_configuration
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 5963
Build 8176: arc lint + arc unit

Event Timeline

ardumont created this revision.May 29 2019, 2:50 PM
Harbormaster completed remote builds in B5958: Diff 5010.May 29 2019, 2:50 PM
ardumont added inline comments.May 29 2019, 2:54 PM
data/defaults.yaml
1899

As discussed earlier, i tried to rename that to swhlister but then i could no longer connect to the db.
When renaming, the instruction mentions it drops the password, even setting it up back (for swhlister), that refuses to connect.

I did not want to lose too much time on it so for now, i moved along.

1979

I dropped the legacy queue names which are no longer used btw.

data/location/sesi_rocquencourt.yaml
29

oops!

ardumont updated this revision to Diff 5011.May 29 2019, 2:57 PM

Do not drop the loader_debian by mistake just yet (oops)

Harbormaster completed remote builds in B5959: Diff 5011.May 29 2019, 2:57 PM
ardumont edited the test plan for this revision. (Show Details)May 29 2019, 2:58 PM
ardumont edited the summary of this revision. (Show Details)May 29 2019, 3:01 PM
ardumont mentioned this in T1772: Lister: Simplify production deployment for listers.
ardumont updated this revision to Diff 5014.May 29 2019, 3:13 PM

Fix missing db's host/port entry configurations

Harbormaster completed remote builds in B5962: Diff 5014.May 29 2019, 3:13 PM
ardumont edited the test plan for this revision. (Show Details)May 29 2019, 3:14 PM
ardumont updated this revision to Diff 5015.May 29 2019, 3:19 PM

Unify the loader-debian configuration entry

This is a bit related as this loader uses the same db as the lister debian.

Harbormaster completed remote builds in B5963: Diff 5015.May 29 2019, 3:19 PM
ardumont updated this revision to Diff 5016.May 29 2019, 3:39 PM

Rebase and plug on production branch

Harbormaster completed remote builds in B5968: Diff 5016.May 29 2019, 3:39 PM
This revision was not accepted when it landed; it landed in state Needs Review.May 29 2019, 3:41 PM
Closed by commit rSPSITE38342bebdef6: swh-lister: Merge lister configuration and service into one (authored by ardumont). · Explain Why
This revision was automatically updated to reflect the committed changes.
ardumont added a comment.May 29 2019, 4:08 PM

Accepted by @douardda orally ;)

Revision Contents
Changeset List

PathSize
data/
151 lines
location/
6 lines
site-modules/
profile/
manifests/
swh/
deploy/
worker/
15 lines

Diff 5015

View Options

data/defaults.yaml

Loading...
View Options

data/location/sesi_rocquencourt.yaml

Loading...
View Options

site-modules/profile/manifests/swh/deploy/base_lister.pp

Loading...
View Options

site-modules/profile/manifests/swh/deploy/worker/lister.pp

Loading...
View Options

site-modules/profile/manifests/swh/deploy/worker/lister_debian.pp

Loading...
View Options

site-modules/profile/manifests/swh/deploy/worker/lister_github.pp

Loading...
View Options

site-modules/profile/manifests/swh/deploy/worker/lister_gitlab.pp

Loading...
View Options

site-modules/profile/manifests/swh/deploy/worker/lister_npm.pp

Loading...
View Options

site-modules/profile/manifests/swh/deploy/worker/lister_pypi.pp

Loading...
About · Developer information · Legal