Details

Reviewers

None

Group Reviewers

System administrators

Maniphest Tasks

T4406: Deploy swh-indexer v2.2 on staging then production

Commits

rSPSITEfde96db56f43: Migrate indexer services to indexer journal client services

Summary

This cleans up old indexer services not based on journal client
(swh-worker@${old}.servicd for old in {indexer_content_mimetype,
indexer_fossology_license}).

It installs new services swh-indexer-journal-client@${new}.service for new in
{mimetype,fossology_license,intrinsic_metadata,extrinsic_metadata).

Related to T4406

Test Plan

vagrant:

root@worker0:~# puppet agent --test
...
Notice: /Stage[main]/Icinga2::Service/Service[icinga2]: Triggered 'refresh' from 1 event
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Base/Profile::Cron::D[cleanup-workers-tmp]/Profile::Cron::File[swh-worker]/File[/etc/puppet-cron.d/swh-worker]/ensure: defined content as '{md5}bfdd5f183ad8b0874184d8c58d094e7c'
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Base/Profile::Cron::D[cleanup-workers-tmp]/Profile::Cron::File[swh-worker]/File[/etc/cron.d/puppet-swh-worker]/ensure: created
Notice: Applied catalog in 4.97 seconds

root@worker0:~# systemctl list-units | grep journal-client
  swh-indexer-journal-client.service                                                        loaded active     running      Software Heritage Indexer Journal Client
  swh-indexer-journal-client@extrinsic_metadata.service                                     loaded activating auto-restart Software Heritage Indexer Journal Client extrinsic_metadata
  swh-indexer-journal-client@fossology_license.service                                      loaded activating auto-restart Software Heritage Indexer Journal Client fossology_license
  swh-indexer-journal-client@intrinsic_metadata.service                                     loaded activating auto-restart Software Heritage Indexer Journal Client intrinsic_metadata
  swh-indexer-journal-client@mimetype.service                                               loaded activating auto-restart Software Heritage Indexer Journal Client mimetype
# for each service ^, parameters.conf defines the necessary override, for example for mimetype
root@worker0:~# cat /etc/systemd/system/swh-indexer-journal-client@mimetype.service.d/parameters.conf
# Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::indexer_journal_client

[Service]
Environment=SWH_CONFIG_FILENAME=/etc/softwareheritage/indexer/mimetype.yml
Environment=SWH_SENTRY_DSN=https://swh::deploy::indexer::sentry_token@sentry.softwareheritage.org/5
Environment=SWH_SENTRY_ENVIRONMENT=staging
Environment=SWH_MAIN_PACKAGE=swh.indexer
Environment=LOGLEVEL=INFO

octo-diff:

$ $SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff worker0.internal.staging.swh.network
Found host worker0.internal.staging.swh.network
...
*******************************************
- Concat_fragment[profile::cron::swh-worker-indexer_content_mimetype-autorestart]
*******************************************
- Concat_fragment[profile::cron::swh-worker-indexer_fossology_license-autorestart]
*******************************************
+ File[/etc/softwareheritage/indexer/extrinsic_metadata.yml] =>
   parameters =>
     "content": "---\nscheduler:\n  cls: remote\n  url: http://scheduler0.interna...
     "ensure": "present",
     "group": "swhworker",
     "mode": "0640",
     "notify": "Service[swh-indexer-journal-client@extrinsic_metadata.service]",
     "owner": "root"
*******************************************
+ File[/etc/softwareheritage/indexer/fossology_license.yml] =>
   parameters =>
     "content": "---\nindexer_storage:\n  cls: remote\n  url: http://storage1.int...
     "ensure": "present",
     "group": "swhworker",
     "mode": "0640",
     "notify": "Service[swh-indexer-journal-client@fossology_license.service]",
     "owner": "root"
*******************************************
+ File[/etc/softwareheritage/indexer/intrinsic_metadata.yml] =>
   parameters =>
     "content": "---\nindexer_storage:\n  cls: remote\n  url: http://storage1.int...
     "ensure": "present",
     "group": "swhworker",
     "mode": "0640",
     "notify": "Service[swh-indexer-journal-client@intrinsic_metadata.service]",
     "owner": "root"
*******************************************
- File[/etc/softwareheritage/indexer/journal_client.yml]
*******************************************
+ File[/etc/softwareheritage/indexer/mimetype.yml] =>
   parameters =>
     "content": "---\nindexer_storage:\n  cls: remote\n  url: http://storage1.int...
     "ensure": "present",
     "group": "swhworker",
     "mode": "0640",
     "notify": "Service[swh-indexer-journal-client@mimetype.service]",
     "owner": "root"
*******************************************
  File[/etc/softwareheritage/indexer_content_mimetype.yml] =>
   parameters =>
     ensure =>
      - present
      + absent
*******************************************
  File[/etc/softwareheritage/indexer_fossology_license.yml] =>
   parameters =>
     ensure =>
      - present
      + absent
*******************************************
- File[/etc/softwareheritage/indexer_origin_intrinsic_metadata.yml]
*******************************************
- File[/etc/systemd/system/swh-indexer-journal-client.service]
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client@.service] =>
   parameters =>
     "content": "# Indexer Journal Client unit file\n# Managed by puppet class pr...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": "Class[Systemd::Systemctl::Daemon_reload]",
     "owner": "root",
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client@extrinsic_metadata.service.d/parameters.conf] =>
   parameters =>
     "content": "# Managed by puppet - modifications will be overwritten\n# In de...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Class[Systemd::Systemctl::Daemon_reload]"
     ],
     "owner": "root",
     "selinux_ignore_defaults": false,
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client@extrinsic_metadata.service.d] =>
   parameters =>
     "ensure": "directory",
     "group": "root",
     "owner": "root",
     "purge": true,
     "recurse": true,
     "selinux_ignore_defaults": false
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client@fossology_license.service.d/parameters.conf] =>
   parameters =>
     "content": "# Managed by puppet - modifications will be overwritten\n# In de...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Class[Systemd::Systemctl::Daemon_reload]"
     ],
     "owner": "root",
     "selinux_ignore_defaults": false,
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client@fossology_license.service.d] =>
   parameters =>
     "ensure": "directory",
     "group": "root",
     "owner": "root",
     "purge": true,
     "recurse": true,
     "selinux_ignore_defaults": false
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client@intrinsic_metadata.service.d/parameters.conf] =>
   parameters =>
     "content": "# Managed by puppet - modifications will be overwritten\n# In de...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Class[Systemd::Systemctl::Daemon_reload]"
     ],
     "owner": "root",
     "selinux_ignore_defaults": false,
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client@intrinsic_metadata.service.d] =>
   parameters =>
     "ensure": "directory",
     "group": "root",
     "owner": "root",
     "purge": true,
     "recurse": true,
     "selinux_ignore_defaults": false
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client@mimetype.service.d/parameters.conf] =>
   parameters =>
     "content": "# Managed by puppet - modifications will be overwritten\n# In de...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Class[Systemd::Systemctl::Daemon_reload]"
     ],
     "owner": "root",
     "selinux_ignore_defaults": false,
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/swh-indexer-journal-client@mimetype.service.d] =>
   parameters =>
     "ensure": "directory",
     "group": "root",
     "owner": "root",
     "purge": true,
     "recurse": true,
     "selinux_ignore_defaults": false
*******************************************
  File[/etc/systemd/system/swh-worker@indexer_content_mimetype.service.d/parameters.conf] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
- File[/etc/systemd/system/swh-worker@indexer_content_mimetype.service.d]
*******************************************
  File[/etc/systemd/system/swh-worker@indexer_fossology_license.service.d/parameters.conf] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
- File[/etc/systemd/system/swh-worker@indexer_fossology_license.service.d]
*******************************************
- File[/etc/systemd/system/swh-worker@indexer_origin_intrinsic_metadata.service.d/parameters.conf]
*******************************************
  Package[python3-swh.indexer] =>
   parameters =>
     notify =>
      - ["Profile::Swh::Deploy::Worker::Instance[indexer_content_mimetype]", "Profile::Swh::Deploy::Worker::Instance[indexer_fossology_license]"]
      + ["Profile::Swh::Deploy::Indexer_journal_client[fossology_license]", "Profile::Swh::Deploy::Indexer_journal_client[mimetype]"]
*******************************************
- Profile::Cron::D[swh-worker-indexer_content_mimetype-autorestart]
*******************************************
- Profile::Cron::D[swh-worker-indexer_fossology_license-autorestart]
*******************************************
+ Profile::Swh::Deploy::Indexer_journal_client[extrinsic_metadata] =>
   parameters =>
     "ensure": "present",
     "instance_name": "extrinsic_metadata",
     "sentry_name": "indexer"
*******************************************
+ Profile::Swh::Deploy::Indexer_journal_client[fossology_license] =>
   parameters =>
     "ensure": "present",
     "instance_name": "fossology_license",
     "sentry_name": "indexer"
*******************************************
+ Profile::Swh::Deploy::Indexer_journal_client[intrinsic_metadata] =>
   parameters =>
     "ensure": "present",
     "instance_name": "intrinsic_metadata",
     "sentry_name": "indexer"
*******************************************
+ Profile::Swh::Deploy::Indexer_journal_client[mimetype] =>
   parameters =>
     "ensure": "present",
     "instance_name": "mimetype",
     "sentry_name": "indexer"
*******************************************
  Profile::Swh::Deploy::Worker::Instance[indexer_content_mimetype] =>
   parameters =>
     ensure =>
      - present
      + absent
     send_task_events =>
      - true
      + false
     sentry_name =>
      - indexer
      + indexer_content_mimetype
*******************************************
  Profile::Swh::Deploy::Worker::Instance[indexer_fossology_license] =>
   parameters =>
     ensure =>
      - present
      + absent
     send_task_events =>
      - true
      + false
     sentry_name =>
      - indexer
      + indexer_fossology_license
*******************************************
- Profile::Swh::Deploy::Worker::Instance[indexer_origin_intrinsic_metadata]
*******************************************
+ Service[swh-indexer-journal-client@extrinsic_metadata.service] =>
   parameters =>
     "enable": true,
     "ensure": "running"
*******************************************
+ Service[swh-indexer-journal-client@fossology_license.service] =>
   parameters =>
     "enable": true,
     "ensure": "running"
*******************************************
+ Service[swh-indexer-journal-client@intrinsic_metadata.service] =>
   parameters =>
     "enable": true,
     "ensure": "running"
*******************************************
+ Service[swh-indexer-journal-client@mimetype.service] =>
   parameters =>
     "enable": true,
     "ensure": "running"
*******************************************
- Service[swh-indexer-journal-client]
*******************************************
  Service[swh-worker@indexer_content_mimetype] =>
   parameters =>
     enable =>
      - true
     ensure =>
      + stopped
*******************************************
  Service[swh-worker@indexer_fossology_license] =>
   parameters =>
     enable =>
      - true
     ensure =>
      + stopped
*******************************************
- Service[swh-worker@indexer_origin_intrinsic_metadata]
*******************************************
+ Systemd::Dropin_file[swh-indexer-journal-client@extrinsic_metadata.service/parameters.conf] =>
   parameters =>
     "content": "# Managed by puppet - modifications will be overwritten\n# In de...
     "daemon_reload": "lazy",
     "ensure": "present",
     "filename": "parameters.conf",
     "group": "root",
     "mode": "0444",
     "owner": "root",
     "path": "/etc/systemd/system",
     "selinux_ignore_defaults": false,
     "show_diff": true,
     "unit": "swh-indexer-journal-client@extrinsic_metadata.service"
*******************************************
+ Systemd::Dropin_file[swh-indexer-journal-client@fossology_license.service/parameters.conf] =>
   parameters =>
     "content": "# Managed by puppet - modifications will be overwritten\n# In de...
     "daemon_reload": "lazy",
     "ensure": "present",
     "filename": "parameters.conf",
     "group": "root",
     "mode": "0444",
     "owner": "root",
     "path": "/etc/systemd/system",
     "selinux_ignore_defaults": false,
     "show_diff": true,
     "unit": "swh-indexer-journal-client@fossology_license.service"
*******************************************
+ Systemd::Dropin_file[swh-indexer-journal-client@intrinsic_metadata.service/parameters.conf] =>
   parameters =>
     "content": "# Managed by puppet - modifications will be overwritten\n# In de...
     "daemon_reload": "lazy",
     "ensure": "present",
     "filename": "parameters.conf",
     "group": "root",
     "mode": "0444",
     "owner": "root",
     "path": "/etc/systemd/system",
     "selinux_ignore_defaults": false,
     "show_diff": true,
     "unit": "swh-indexer-journal-client@intrinsic_metadata.service"
*******************************************
+ Systemd::Dropin_file[swh-indexer-journal-client@mimetype.service/parameters.conf] =>
   parameters =>
     "content": "# Managed by puppet - modifications will be overwritten\n# In de...
     "daemon_reload": "lazy",
     "ensure": "present",
     "filename": "parameters.conf",
     "group": "root",
     "mode": "0444",
     "owner": "root",
     "path": "/etc/systemd/system",
     "selinux_ignore_defaults": false,
     "show_diff": true,
     "unit": "swh-indexer-journal-client@mimetype.service"
*******************************************
  Systemd::Dropin_file[swh-worker@indexer_content_mimetype/parameters.conf] =>
   parameters =>
     content =>
      - # Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=1
Environment=MAX_TASKS_PER_CHILD=5
Environment=LOGLEVEL=info
Environment=SWH_SENTRY_DSN=https://swh::deploy::indexer::sentry_token@sentry.softwareheritage.org/5
Environment=SWH_SENTRY_ENVIRONMENT=staging
Environment=SWH_MAIN_PACKAGE=swh.indexer


Environment=SWH_WORKER_CLI_EXTRA_ARGS=--events

     ensure =>
      - present
      + absent
*******************************************
  Systemd::Dropin_file[swh-worker@indexer_fossology_license/parameters.conf] =>
   parameters =>
     content =>
      - # Managed by puppet - modifications will be overwritten
# In defined class profile::swh::deploy::worker::instance

[Service]
Environment=CONCURRENCY=1
Environment=MAX_TASKS_PER_CHILD=5
Environment=LOGLEVEL=info
Environment=SWH_SENTRY_DSN=https://swh::deploy::indexer::sentry_token@sentry.softwareheritage.org/5
Environment=SWH_SENTRY_ENVIRONMENT=staging
Environment=SWH_MAIN_PACKAGE=swh.indexer


Environment=SWH_WORKER_CLI_EXTRA_ARGS=--events

     ensure =>
      - present
      + absent
*******************************************
- Systemd::Dropin_file[swh-worker@indexer_origin_intrinsic_metadata/parameters.conf]
*******************************************
- Systemd::Unit_file[swh-indexer-journal-client.service]
*******************************************
+ Systemd::Unit_file[swh-indexer-journal-client@.service] =>
   parameters =>
     "content": "# Indexer Journal Client unit file\n# Managed by puppet class pr...
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true
*******************************************
*** End octocatalog-diff on worker0.internal.staging.swh.network

Diff Detail

Repository

rSPSITE puppet-swh-site

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Jul 27 2022, 12:01 PM

Herald added a reviewer: System administrators. · View Herald TranscriptJul 27 2022, 12:01 PM

Harbormaster completed remote builds in B30548: Diff 29474.Jul 27 2022, 12:01 PM

ardumont requested review of this revision.Jul 27 2022, 12:01 PM

ardumont edited the summary of this revision. (Show Details)Jul 27 2022, 12:05 PM

ardumont edited the test plan for this revision. (Show Details)

ardumont mentioned this in T4406: Deploy swh-indexer v2.2 on staging then production.Jul 27 2022, 12:08 PM

Fix typo in actual service running the journal client

Harbormaster completed remote builds in B30549: Diff 29475.Jul 27 2022, 12:13 PM

ardumont edited the test plan for this revision. (Show Details)Jul 27 2022, 12:23 PM

ardumont edited the test plan for this revision. (Show Details)

Rebase

Harbormaster completed remote builds in B30550: Diff 29476.Jul 28 2022, 12:38 PM

Rebase

Harbormaster completed remote builds in B30557: Diff 29483.Jul 29 2022, 10:25 AM

This revision was not accepted when it landed; it landed in state Needs Review.Jul 29 2022, 10:27 AM

Closed by commit rSPSITEfde96db56f43: Migrate indexer services to indexer journal client services (authored by ardumont). · Explain Why

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rSPSITEfde96db56f43: Migrate indexer services to indexer journal client services.

Migrate indexer services to indexer journal client services
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 29484

data/common/common.yaml

data/deployments/production/azure_euwest.yaml

data/deployments/staging/common.yaml

site-modules/profile/manifests/swh/deploy/base_indexer.pp

site-modules/profile/manifests/swh/deploy/indexer_journal_client.pp

site-modules/profile/manifests/swh/deploy/worker/indexer_content_mimetype.pp

site-modules/profile/manifests/swh/deploy/worker/indexer_fossology_license.pp

site-modules/profile/manifests/swh/deploy/worker/indexer_origin_extrinsic_metadata.pp

site-modules/profile/manifests/swh/deploy/worker/indexer_origin_intrinsic_metadata.pp

site-modules/profile/templates/swh/deploy/journal/parameters.conf.erb

site-modules/profile/templates/swh/deploy/journal/swh-indexer-journal-client.service.erb

site-modules/profile/templates/swh/deploy/journal/swh-indexer-journal-client@.service.erb

Migrate indexer services to indexer journal client servicesClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 29484

data/common/common.yaml

data/deployments/production/azure_euwest.yaml

data/deployments/staging/common.yaml

site-modules/profile/manifests/swh/deploy/base_indexer.pp

site-modules/profile/manifests/swh/deploy/indexer_journal_client.pp

site-modules/profile/manifests/swh/deploy/worker/indexer_content_mimetype.pp

site-modules/profile/manifests/swh/deploy/worker/indexer_fossology_license.pp

site-modules/profile/manifests/swh/deploy/worker/indexer_origin_extrinsic_metadata.pp

site-modules/profile/manifests/swh/deploy/worker/indexer_origin_intrinsic_metadata.pp

site-modules/profile/templates/swh/deploy/journal/parameters.conf.erb

site-modules/profile/templates/swh/deploy/journal/swh-indexer-journal-client.service.erb

site-modules/profile/templates/swh/deploy/journal/swh-indexer-journal-client@.service.erb

Migrate indexer services to indexer journal client services
ClosedPublic
Actions

Revision Contents
Changeset List