Page MenuHomeSoftware Heritage

Dedicate a loader_oneshot service for temporary use
ClosedPublic

Authored by ardumont on Jun 2 2021, 5:36 PM.

Details

Summary

This will be used for the first sourceforge ingestion for example.

Another dedicated commit is refactoring to avoid the max content size setup
duplication (shown alongside the diff as 2nd commit)

Related to T3350

Test Plan

octocatalog-diff:

bin/octocatalog-diff --to staging worker17
Found host worker17.softwareheritage.org
WARN     -> Environment "wip-pg-hba-rules-in-yaml" contained non-word characters, correcting name to wip_pg_hba_rules_in_yaml
Cloning into '/tmp/swh-ocd.8ga1hIwd/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.8ga1hIwd/environments/staging/data/private'...
done.
*** Running octocatalog-diff on host worker17.softwareheritage.org
I, [2021-06-02T17:35:42.298927 #7298]  INFO -- : Catalogs compiled for worker17.softwareheritage.org
I, [2021-06-02T17:35:43.137767 #7298]  INFO -- : Diffs computed for worker17.softwareheritage.org
diff origin/production/worker17.softwareheritage.org current/worker17.softwareheritage.org
*******************************************
+ Concat_file[profile::cron::swh-worker] =>
   parameters =>
     "group": "root",
     "mode": "0644",
     "owner": "root",
     "path": "/etc/puppet-cron.d/swh-worker",
     "tag": "profile::cron::swh-worker"
*******************************************
+ Concat_fragment[profile::cron::cleanup-workers-tmp] =>
   parameters =>
     "content": "# Cron snippet cleanup-workers-tmp\n38 0-22/2 * * * root find /t...
     "order": "10",
     "tag": "profile::cron::swh-worker",
     "target": "profile::cron::swh-worker"
*******************************************
+ Concat_fragment[profile::cron::swh-worker-loader_oneshot-autorestart] =>
   parameters =>
     "content": "# Cron snippet swh-worker-loader_oneshot-autorestart\n0-45/15 * ...
     "order": "10",
     "tag": "profile::cron::swh-worker",
     "target": "profile::cron::swh-worker"
*******************************************
+ Concat_fragment[profile::cron::swh-worker::_header] =>
   parameters =>
     "content": "# Managed by puppet (module profile::cron), manual changes will ...
     "order": "00",
     "tag": "profile::cron::swh-worker",
     "target": "profile::cron::swh-worker"
*******************************************
+ File[/etc/cron.d/puppet-swh-worker] =>
   parameters =>
     "ensure": "link",
     "target": "/etc/puppet-cron.d/swh-worker"
*******************************************
+ File[/etc/softwareheritage/loader] =>
   parameters =>
     "ensure": "directory",
     "group": "swhworker",
     "mode": "0644",
     "owner": "swhworker"
*******************************************
+ File[/etc/softwareheritage/loader_oneshot.yml] =>
   parameters =>
     "content": "---\nstorage:\n  cls: pipeline\n  steps:\n  - cls: buffer\n    m...
     "ensure": "present",
     "group": "swhworker",
     "mode": "0640",
     "owner": "swhworker"
*******************************************
+ File[/etc/systemd/system/swh-worker.service] =>
   parameters =>
     "content": "[Unit]\nDescription=Software Heritage Worker (meta-task)\n\n[Ser...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": "Class[Systemd::Systemctl::Daemon_reload]",
     "owner": "root",
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/swh-worker@.service] =>
   parameters =>
     "content": "[Unit]\nDescription=Software Heritage Worker (%i)\nAfter=network...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": "Class[Systemd::Systemctl::Daemon_reload]",
     "owner": "root",
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/swh-worker@loader_oneshot.service.d/parameters.conf] =>
   parameters =>
     "content": "# Managed by puppet - modifications will be overwritten\n# In de...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Class[Systemd::Systemctl::Daemon_reload]"
     ],
     "owner": "root",
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/swh-worker@loader_oneshot.service.d] =>
   parameters =>
     "ensure": "directory",
     "group": "root",
     "owner": "root",
     "purge": true,
     "recurse": true
*******************************************
+ File[/etc/systemd/system/system-swh\x2dworker.slice] =>
   parameters =>
     "content": "[Unit]\nDescription=SWH workers\n\n[Slice]\nMemoryMax=90%\n",
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": "Class[Systemd::Systemctl::Daemon_reload]",
     "owner": "root",
     "show_diff": true
*******************************************
+ File[/usr/local/sbin/swh-worker-ping-restart] =>
   parameters =>
     "content": "#!/bin/bash\n\nset -e\n\nis_service_enabled () {\n\tservice_name...
     "group": "root",
     "mode": "0755",
     "owner": "root"
*******************************************
+ Package[python3-swh.loader.git] =>
   parameters =>
     "ensure": "present"
*******************************************
+ Package[python3-swh.loader.mercurial] =>
   parameters =>
     "ensure": "present"
*******************************************
+ Package[python3-swh.loader.svn] =>
   parameters =>
     "ensure": "present"
*******************************************
+ Package[python3-swh.scheduler] =>
   parameters =>
     "ensure": "installed"
*******************************************
+ Profile::Cron::D[cleanup-workers-tmp] =>
   parameters =>
     "command": "find /tmp -depth -mindepth 3 -maxdepth 3 -type d -ctime +2 -exec...
     "hour": "fqdn_rand/2",
     "minute": "fqdn_rand",
     "target": "swh-worker",
     "unique_tag": "cleanup-workers-tmp",
     "user": "root"
*******************************************
+ Profile::Cron::D[swh-worker-loader_oneshot-autorestart] =>
   parameters =>
     "command": "chronic /usr/local/sbin/swh-worker-ping-restart loader_oneshot@w...
     "minute": "fqdn_rand/15",
     "target": "swh-worker",
     "unique_tag": "swh-worker-loader_oneshot-autorestart",
     "user": "root"
*******************************************
+ Profile::Cron::File[swh-worker] =>
   parameters =>
     "target": "swh-worker"
*******************************************
+ Profile::Swh::Deploy::Worker::Instance[loader_oneshot] =>
   parameters =>
     "ensure": "present",
     "instance_name": "loader_oneshot",
     "merge_policy": "deep",
     "sentry_name": "loader_oneshot"
*******************************************
+ Service[swh-worker@loader_oneshot] =>
   parameters =>
     "enable": true
*******************************************
+ Service[swh-worker] =>
   parameters =>
     "enable": true,
     "ensure": "running"
*******************************************
+ Systemd::Dropin_file[swh-worker@loader_oneshot/parameters.conf] =>
   parameters =>
     "content": "# Managed by puppet - modifications will be overwritten\n# In de...
     "daemon_reload": "lazy",
     "ensure": "present",
     "filename": "parameters.conf",
     "group": "root",
     "mode": "0444",
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true,
     "unit": "swh-worker@loader_oneshot.service"
*******************************************
+ Systemd::Unit_file[swh-worker.service] =>
   parameters =>
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Service[swh-worker]"
     ],
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true,
     "source": "puppet:///modules/profile/swh/deploy/worker/swh-worker.service"
*******************************************
+ Systemd::Unit_file[swh-worker@.service] =>
   parameters =>
     "content": "[Unit]\nDescription=Software Heritage Worker (%i)\nAfter=network...
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true
*******************************************
+ Systemd::Unit_file[system-swh\x2dworker.slice] =>
   parameters =>
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true,
     "source": "puppet:///modules/profile/swh/deploy/worker/system-swh\\x2dworker...
*******************************************
*** End octocatalog-diff on worker17.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.
  • Bump the concurrency to 4
  • Expose the other commits which avoids max content size duplication
ardumont added inline comments.
data/hostname/worker17.softwareheritage.org.yaml
3

could be 8 but i'm starting slow.

This revision is now accepted and ready to land.Jun 3 2021, 10:20 AM