Page MenuHomeSoftware Heritage

Move indexer_journal_client to scheduler instances
ClosedPublic

Authored by ardumont on Feb 10 2021, 5:24 PM.

Details

Summary

No need for those journal client to be on kafka nodes.

This is also an attempt to clarify where to deploy our backfill configuration.

In effect, this moves the service from:

  • prod: getty -> saatchi
  • staging: journal0 -> scheduler0

Deployment:

  • deactivate puppet on saatchi, scheduler0
  • stop old services on getty and journal0
  • (clean up)
  • puppet run on saatchi, scheduler0
Test Plan

octocatalog-diff:

$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging getty
...
I, [2021-02-10T17:09:55.240238 #14284]  INFO -- : Catalogs compiled for getty.internal.softwareheritage.org
I, [2021-02-10T17:09:56.576597 #14284]  INFO -- : Diffs computed for getty.internal.softwareheritage.org
diff origin/production/getty.internal.softwareheritage.org current/getty.internal.softwareheritage.org
*******************************************
- File[/etc/softwareheritage/indexer/journal_client.yml]
*******************************************
- File[/etc/softwareheritage/indexer]
*******************************************
- File[/etc/softwareheritage/journal]
*******************************************
- File[/etc/systemd/system/swh-indexer-journal-client.service]
*******************************************
- Package[python3-swh.indexer]
*******************************************
- Package[python3-swh.journal]
*******************************************
- Service[swh-indexer-journal-client]
*******************************************
- Systemd::Unit_file[swh-indexer-journal-client.service]
*******************************************
*** End octocatalog-diff on getty.internal.softwareheritage.org
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging journal0.internal.staging.swh.network
...
*** Running octocatalog-diff on host journal0.internal.staging.swh.network
I, [2021-02-10T17:10:26.109041 #19467]  INFO -- : Catalogs compiled for journal0.internal.staging.swh.network
I, [2021-02-10T17:10:27.498780 #19467]  INFO -- : Diffs computed for journal0.internal.staging.swh.network
diff origin/production/journal0.internal.staging.swh.network current/journal0.internal.staging.swh.network
*******************************************
- File[/etc/softwareheritage/indexer/journal_client.yml]
*******************************************
- File[/etc/softwareheritage/indexer]
*******************************************
- File[/etc/softwareheritage/journal]
*******************************************
- File[/etc/systemd/system/swh-indexer-journal-client.service]
*******************************************
- Package[python3-swh.indexer]
*******************************************
- Package[python3-swh.journal]
*******************************************
- Service[swh-indexer-journal-client]
*******************************************
- Systemd::Unit_file[swh-indexer-journal-client.service]
*******************************************
*** End octocatalog-diff on journal0.internal.staging.swh.network
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging scheduler0.internal.staging.swh.network
...
*** Running octocatalog-diff on host scheduler0.internal.staging.swh.network
I, [2021-02-10T17:10:57.460041 #24611]  INFO -- : Catalogs compiled for scheduler0.internal.staging.swh.network
I, [2021-02-10T17:10:59.194738 #24611]  INFO -- : Diffs computed for scheduler0.internal.staging.swh.network
diff origin/production/scheduler0.internal.staging.swh.network current/scheduler0.internal.staging.swh.network
*******************************************
+ File[/etc/softwareheritage/indexer/journal_client.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhdev"
      "mode": "0644"
      "notify": "Service[swh-indexer-journal-client]"
      "owner": "root"
      "content": >>>
---
journal:
  brokers:
  - journal0.internal.staging.swh.network
  group_id: swh.indexer.journal_client
  prefix: swh.journal.objects
scheduler:
  cls: remote
  args:
    url: http://scheduler0.internal.staging.swh.network:5008/
<<<
*******************************************
+ File[/etc/softwareheritage/indexer] =>
   parameters =>
      "ensure": "directory"
      "group": "swhworker"
      "mode": "0755"
      "owner": "swhworker"
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client.service] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": "Class[Systemd::Systemctl::Daemon_reload]"
      "owner": "root"
      "show_diff": true
      "content": >>>
# Indexer Journal Client unit file
# Managed by puppet class profile::swh::deploy::indexer_journal_client
# Changes will be overwritten

[Unit]
Description=Software Heritage Indexer Journal Client
After=network.target

[Service]
User=swhstorage
Group=swhstorage
Type=simple
ExecStart=/usr/bin/swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
<<<
*******************************************
+ Package[python3-swh.indexer] =>
   parameters =>
      "ensure": "present"
*******************************************
+ Service[swh-indexer-journal-client] =>
   parameters =>
      "enable": true
      "ensure": "running"
*******************************************
+ Systemd::Unit_file[swh-indexer-journal-client.service] =>
   parameters =>
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "notify": ["Service[swh-indexer-journal-client]"]
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
# Indexer Journal Client unit file
# Managed by puppet class profile::swh::deploy::indexer_journal_client
# Changes will be overwritten

[Unit]
Description=Software Heritage Indexer Journal Client
After=network.target

[Service]
User=swhstorage
Group=swhstorage
Type=simple
ExecStart=/usr/bin/swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
<<<
*******************************************
*** End octocatalog-diff on scheduler0.internal.staging.swh.network
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging saatchi
...
*** Running octocatalog-diff on host saatchi.internal.softwareheritage.org
I, [2021-02-10T17:15:05.562029 #30505]  INFO -- : Catalogs compiled for saatchi.internal.softwareheritage.org
I, [2021-02-10T17:15:07.186209 #30505]  INFO -- : Diffs computed for saatchi.internal.softwareheritage.org
diff origin/production/saatchi.internal.softwareheritage.org current/saatchi.internal.softwareheritage.org
*******************************************
+ File[/etc/softwareheritage/indexer/journal_client.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhdev"
      "mode": "0644"
      "notify": "Service[swh-indexer-journal-client]"
      "owner": "root"
      "content": >>>
---
journal:
  brokers:
  - kafka1.internal.softwareheritage.org
  - kafka2.internal.softwareheritage.org
  - kafka3.internal.softwareheritage.org
  - kafka4.internal.softwareheritage.org
  group_id: swh.indexer.journal_client
  prefix: swh.journal.objects
scheduler:
  cls: remote
  url: http://saatchi.internal.softwareheritage.org:5008/
<<<
*******************************************
+ File[/etc/softwareheritage/indexer] =>
   parameters =>
      "ensure": "directory"
      "group": "swhworker"
      "mode": "0755"
      "owner": "swhworker"
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client.service] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": "Class[Systemd::Systemctl::Daemon_reload]"
      "owner": "root"
      "show_diff": true
      "content": >>>
# Indexer Journal Client unit file
# Managed by puppet class profile::swh::deploy::indexer_journal_client
# Changes will be overwritten

[Unit]
Description=Software Heritage Indexer Journal Client
After=network.target

[Service]
User=swhstorage
Group=swhstorage
Type=simple
ExecStart=/usr/bin/swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
<<<
*******************************************
+ Package[python3-swh.indexer] =>
   parameters =>
      "ensure": "present"
*******************************************
+ Service[swh-indexer-journal-client] =>
   parameters =>
      "enable": true
      "ensure": "running"
*******************************************
+ Systemd::Unit_file[swh-indexer-journal-client.service] =>
   parameters =>
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "notify": ["Service[swh-indexer-journal-client]"]
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
# Indexer Journal Client unit file
# Managed by puppet class profile::swh::deploy::indexer_journal_client
# Changes will be overwritten

[Unit]
Description=Software Heritage Indexer Journal Client
After=network.target

[Service]
User=swhstorage
Group=swhstorage
Type=simple
ExecStart=/usr/bin/swh indexer --config-file /etc/softwareheritage/indexer/journal_client.yml journal-client
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
<<<
*******************************************
*** End octocatalog-diff on saatchi.internal.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.

LGTM but it's biased as we worked onit together ;)

This revision is now accepted and ready to land.Feb 10 2021, 5:30 PM