Page MenuHomeSoftware Heritage
Differential D7055

storage01.euwest.azure: Only keep the gunicorn-swh-storage service
ClosedPublic
Actions

Authored by ardumont on Feb 1 2022, 12:19 PM.

Details

Reviewers
vsellier
Group Reviewers
System administrators
Maniphest Tasks
T3901: clean up storage01.euwest.azure unused services
Commits
rSPSITE14a9832c5a83: storage01.euwest.azure: Only keep the gunicorn-swh-storage service
Summary

The other services installed are unused. It still requires the objstorage-cloud profile
though since that storage services uses the multiplexer objstorage (which includes the
azure blobstorage access).

Related to T3901

Test Plan

This only keeps the gunicorn-swh-storage service and wants to drop the rest (from the
diff) [1]:

$ $SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff storage01.euwest.azure
...
*******************************************
- Concat::Fragment[nginx-swh-indexer-storage-500-6666cd76f96956469e7be39d750cc7d9]
*******************************************
- Concat::Fragment[nginx-swh-indexer-storage-default-500-6666cd76f96956469e7be39d750cc7d9]
*******************************************
- Concat::Fragment[nginx-swh-indexer-storage-default-footer]
*******************************************
- Concat::Fragment[nginx-swh-indexer-storage-default-header]
*******************************************
- Concat::Fragment[nginx-swh-indexer-storage-footer]
*******************************************
- Concat::Fragment[nginx-swh-indexer-storage-header]
*******************************************
- Concat::Fragment[nginx-swh-objstorage-500-6666cd76f96956469e7be39d750cc7d9]
*******************************************
- Concat::Fragment[nginx-swh-objstorage-default-500-6666cd76f96956469e7be39d750cc7d9]
*******************************************
- Concat::Fragment[nginx-swh-objstorage-default-footer]
*******************************************
- Concat::Fragment[nginx-swh-objstorage-default-header]
*******************************************
- Concat::Fragment[nginx-swh-objstorage-footer]
*******************************************
- Concat::Fragment[nginx-swh-objstorage-header]
*******************************************
- Concat::Fragment[swh-indexer-storage-gunicorn_upstream_footer]
*******************************************
- Concat::Fragment[swh-indexer-storage-gunicorn_upstream_header]
*******************************************
- Concat::Fragment[swh-indexer-storage-gunicorn_upstream_member_gunicorn-indexer-storage]
*******************************************
- Concat::Fragment[swh-objstorage-gunicorn_upstream_footer]
*******************************************
- Concat::Fragment[swh-objstorage-gunicorn_upstream_header]
*******************************************
- Concat::Fragment[swh-objstorage-gunicorn_upstream_member_gunicorn-objstorage]
*******************************************
- Concat[/etc/nginx/conf.d/swh-indexer-storage-gunicorn-upstream.conf]
*******************************************
- Concat[/etc/nginx/conf.d/swh-objstorage-gunicorn-upstream.conf]
*******************************************
- Concat[/etc/nginx/sites-available/nginx-swh-indexer-storage-default.conf]
*******************************************
- Concat[/etc/nginx/sites-available/nginx-swh-indexer-storage.conf]
*******************************************
- Concat[/etc/nginx/sites-available/nginx-swh-objstorage-default.conf]
*******************************************
- Concat[/etc/nginx/sites-available/nginx-swh-objstorage.conf]
*******************************************
- Concat_file[/etc/nginx/conf.d/swh-indexer-storage-gunicorn-upstream.conf]
*******************************************
- Concat_file[/etc/nginx/conf.d/swh-objstorage-gunicorn-upstream.conf]
*******************************************
- Concat_file[/etc/nginx/sites-available/nginx-swh-indexer-storage-default.conf]
*******************************************
- Concat_file[/etc/nginx/sites-available/nginx-swh-indexer-storage.conf]
*******************************************
- Concat_file[/etc/nginx/sites-available/nginx-swh-objstorage-default.conf]
*******************************************
- Concat_file[/etc/nginx/sites-available/nginx-swh-objstorage.conf]
*******************************************
- Concat_file[profile::cron::borgmatic]
*******************************************
- Concat_fragment[nginx-swh-indexer-storage-500-6666cd76f96956469e7be39d750cc7d9]
*******************************************
- Concat_fragment[nginx-swh-indexer-storage-default-500-6666cd76f96956469e7be39d750cc7d9]
*******************************************
- Concat_fragment[nginx-swh-indexer-storage-default-footer]
*******************************************
- Concat_fragment[nginx-swh-indexer-storage-default-header]
*******************************************
- Concat_fragment[nginx-swh-indexer-storage-footer]
*******************************************
- Concat_fragment[nginx-swh-indexer-storage-header]
*******************************************
- Concat_fragment[nginx-swh-objstorage-500-6666cd76f96956469e7be39d750cc7d9]
*******************************************
- Concat_fragment[nginx-swh-objstorage-default-500-6666cd76f96956469e7be39d750cc7d9]
*******************************************
- Concat_fragment[nginx-swh-objstorage-default-footer]
*******************************************
- Concat_fragment[nginx-swh-objstorage-default-header]
*******************************************
- Concat_fragment[nginx-swh-objstorage-footer]
*******************************************
- Concat_fragment[nginx-swh-objstorage-header]
*******************************************
- Concat_fragment[profile::cron::borgmatic-create]
*******************************************
- Concat_fragment[profile::cron::borgmatic-full]
*******************************************
- Concat_fragment[profile::cron::borgmatic::_header]
*******************************************
- Concat_fragment[swh-indexer-storage-gunicorn_upstream_footer]
*******************************************
- Concat_fragment[swh-indexer-storage-gunicorn_upstream_header]
*******************************************
- Concat_fragment[swh-indexer-storage-gunicorn_upstream_member_gunicorn-indexer-storage]
*******************************************
- Concat_fragment[swh-objstorage-gunicorn_upstream_footer]
*******************************************
- Concat_fragment[swh-objstorage-gunicorn_upstream_header]
*******************************************
- Concat_fragment[swh-objstorage-gunicorn_upstream_member_gunicorn-objstorage]
*******************************************
- Exec[sed -e /dar\./d -e /swh-dar-backup/d -i /var/spool/cron/crontabs/root]
*******************************************
- Exec[ssh-keygen -t ed25519 -f /root/.ssh/id_ed25519.borg -N '']
*******************************************
- File[/etc/borgmatic/config.yaml]
*******************************************
- File[/etc/borgmatic]
*******************************************
- File[/etc/cron.d/puppet-borgmatic]
*******************************************
- File[/etc/gunicorn/instances/swh-indexer-storage.cfg]
*******************************************
- File[/etc/gunicorn/instances/swh-objstorage.cfg]
*******************************************
- File[/etc/logrotate.d/swh-dar]
*******************************************
- File[/etc/nginx/sites-enabled/nginx-swh-indexer-storage-default.conf]
*******************************************
- File[/etc/nginx/sites-enabled/nginx-swh-indexer-storage.conf]
*******************************************
- File[/etc/nginx/sites-enabled/nginx-swh-objstorage-default.conf]
*******************************************
- File[/etc/nginx/sites-enabled/nginx-swh-objstorage.conf]
*******************************************
- File[/etc/softwareheritage/objstorage/server.yml]
*******************************************
- File[/etc/softwareheritage/objstorage]
*******************************************
- File[/etc/softwareheritage/storage/indexer.yml]
*******************************************
- File[/etc/systemd/system/gunicorn-swh-indexer-storage.service]
*******************************************
- File[/etc/systemd/system/gunicorn-swh-objstorage.service]
*******************************************
- File[/etc/tmpfiles.d/gunicorn-swh-indexer-storage.conf]
*******************************************
- File[/etc/tmpfiles.d/gunicorn-swh-objstorage.conf]
*******************************************
- File[/srv/backups]
*******************************************
- File[/usr/local/bin/swh-dar-backup]
*******************************************
- File[/var/lib/borg]
*******************************************
- File[/var/log/dar]
*******************************************
- Gunicorn::Instance[swh-indexer-storage]
*******************************************
- Gunicorn::Instance[swh-objstorage]
*******************************************
- Nginx::Resource::Location[nginx-swh-indexer-storage-default-default]
*******************************************
- Nginx::Resource::Location[nginx-swh-indexer-storage-default]
*******************************************
- Nginx::Resource::Location[nginx-swh-objstorage-default-default]
*******************************************
- Nginx::Resource::Location[nginx-swh-objstorage-default]
*******************************************
- Nginx::Resource::Server[nginx-swh-indexer-storage-default]
*******************************************
- Nginx::Resource::Server[nginx-swh-indexer-storage]
*******************************************
- Nginx::Resource::Server[nginx-swh-objstorage-default]
*******************************************
- Nginx::Resource::Server[nginx-swh-objstorage]
*******************************************
- Nginx::Resource::Upstream::Member[gunicorn-indexer-storage]
*******************************************
- Nginx::Resource::Upstream::Member[gunicorn-objstorage]
*******************************************
- Nginx::Resource::Upstream[swh-indexer-storage-gunicorn]
*******************************************
- Nginx::Resource::Upstream[swh-objstorage-gunicorn]
*******************************************
- Package[borgbackup]
*******************************************
- Package[borgmatic]
*******************************************
- Package[python3-swh.indexer.storage]
*******************************************
- Package[python3-swh.objstorage]
*******************************************
- Profile::Cron::D[borgmatic-create]
*******************************************
- Profile::Cron::D[borgmatic-full]
*******************************************
- Profile::Cron::File[borgmatic]
*******************************************
- Profile::Swh::Deploy::Rpc_server[indexer-storage]
*******************************************
- Profile::Swh::Deploy::Rpc_server[objstorage]
*******************************************
- Service[gunicorn-swh-indexer-storage]
*******************************************
- Service[gunicorn-swh-objstorage]
*******************************************
- Systemd::Tmpfile[gunicorn-swh-indexer-storage.conf]
*******************************************
- Systemd::Tmpfile[gunicorn-swh-objstorage.conf]
*******************************************
- Systemd::Unit_file[gunicorn-swh-indexer-storage.service]
*******************************************
- Systemd::Unit_file[gunicorn-swh-objstorage.service]
*******************************************
*** End octocatalog-diff on storage01.euwest.azure.internal.softwareheritage.org

The actual service clean up will happen manually on that node.

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Feb 1 2022, 12:19 PM
Herald added a reviewer: System administrators. · View Herald TranscriptFeb 1 2022, 12:19 PM
ardumont requested review of this revision.Feb 1 2022, 12:19 PM
Harbormaster completed remote builds in B26418: Diff 25587.Feb 1 2022, 12:19 PM
ardumont updated this revision to Diff 25588.Feb 1 2022, 12:21 PM

Amend commit message

Harbormaster completed remote builds in B26419: Diff 25588.Feb 1 2022, 12:21 PM
ardumont retitled this revision from Migrate storage01.euwest.azure as a simple storage to storage01.euwest.azure: Only keep the gunicorn-swh-storage service.Feb 1 2022, 12:21 PM
ardumont edited the summary of this revision. (Show Details)
olasd added a subscriber: olasd.Feb 1 2022, 12:51 PM
olasd added inline comments.
site-modules/role/manifests/swh_storage_cloud.pp
1–2

We can probably go all the way to swh_base (and drop the useless backups for this fully puppet-based server)?

ardumont added inline comments.Feb 1 2022, 2:04 PM
site-modules/role/manifests/swh_storage_cloud.pp
1–2

good idea, lemme check that ;)

ardumont updated this revision to Diff 25592.Feb 1 2022, 2:06 PM

Apply suggestion (use swh_base as base class instead of swh_server)

Harbormaster completed remote builds in B26423: Diff 25592.Feb 1 2022, 2:06 PM
ardumont edited the test plan for this revision. (Show Details)Feb 1 2022, 2:06 PM
ardumont marked an inline comment as done.
ardumont added inline comments.
site-modules/role/manifests/swh_storage_cloud.pp
1–2

yep, it works, amended (test plan as well).

ardumont updated this revision to Diff 25593.Feb 1 2022, 2:14 PM

Drop unused configuration

Harbormaster completed remote builds in B26424: Diff 25593.Feb 1 2022, 2:14 PM
vsellier accepted this revision.Feb 1 2022, 2:31 PM
This revision is now accepted and ready to land.Feb 1 2022, 2:31 PM
ardumont updated this revision to Diff 25596.Feb 1 2022, 2:34 PM

Rebase

Harbormaster completed remote builds in B26427: Diff 25596.Feb 1 2022, 2:34 PM
Closed by commit rSPSITE14a9832c5a83: storage01.euwest.azure: Only keep the gunicorn-swh-storage service (authored by ardumont). · Explain WhyFeb 1 2022, 2:34 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rSPSITE14a9832c5a83: storage01.euwest.azure: Only keep the gunicorn-swh-storage service.

Revision Contents
Changeset List

PathSize
data/
hostname/
19 lines
site-modules/
role/
manifests/
3 lines

Diff 25597

View Options

data/hostname/storage01.euwest.azure.internal.softwareheritage.org.yaml

Loading...
View Options

site-modules/role/manifests/swh_storage_cloud.pp

Loading...
About · Developer information · Legal