Details

Reviewers

olasd

Group Reviewers

System administrators

Maniphest Tasks

T3826: Make opam shared root initialization more robust

Commits

rSPSITE791085c55b39: Make opam shared root initialization more robust

Summary

This:

drops the previous version of the tooling
installs the new tooling:
- use set -e in the main script so command failures are immediately detected by icinga
- the script triggered by the timer/service script updates all instances at once (in working order, first opam root with the default opam instance, then the other secondary repositories). Implementation wise, it's looping over all instances to init or update them all.
also changes the timer period to ~24h (this matches the scheduler configuration for

the opam lister)

Related to T3826

Test Plan

*******************************************
- File[/etc/systemd/system/opam-manage-shared-state-coq.inria.fr.service]
*******************************************
  File[/etc/systemd/system/opam-manage-shared-state-coq.inria.fr.timer] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
- File[/etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.service]
*******************************************
  File[/etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.timer] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
- File[/etc/systemd/system/opam-manage-shared-state-opam.ocaml.org.service]
*******************************************
  File[/etc/systemd/system/opam-manage-shared-state-opam.ocaml.org.timer] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
+ File[/etc/systemd/system/opam-manage-shared-state.service] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": "Class[Systemd::Systemctl::Daemon_reload]"
      "owner": "root"
      "show_diff": true
      "content": >>>
# Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state
Requires=network.target
Before=swh-worker@lister.service swh-worker@loader_opam.service
After=network.target

[Service]
User=swhworker
Group=swhworker
Type=oneshot
ExecStart=/usr/local/bin/opam-manage-shared-state.sh

[Install]
WantedBy=swh-worker@lister.service swh-worker@loader_opam.service
<<<
*******************************************
+ File[/etc/systemd/system/opam-manage-shared-state.timer] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": ["Class[Systemd::Systemctl::Daemon_reload]", "Service[opam-manage-shared-state.timer]"]
      "owner": "root"
      "show_diff": true
      "content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
Persistent=true
OnCalendar=daily
RandomizedDelaySec=24h
FixedRandomDelay=true
Unit=.service

[Unit]
Description=Software Heritage Manage OPAM shared state timer
<<<
*******************************************
+ File[/opam-manage-shared-state-coq.inria.fr.service] =>
   parameters =>
      "ensure": "absent"
*******************************************
+ File[/opam-manage-shared-state-ocamlbench-repo.service] =>
   parameters =>
      "ensure": "absent"
*******************************************
+ File[/opam-manage-shared-state-opam.ocaml.org.service] =>
   parameters =>
      "ensure": "absent"
*******************************************
  File[/usr/local/bin/opam-manage-shared-state.sh] =>
   parameters =>
     content =>
      @@ -4,19 +4,50 @@
       # Changes will be overwritten
      _
      -CMD=/usr/bin/opam
      +# This script is in charge of bootstraping the opam root folder with the default opam
      +# repository.
      +
      +set -e
      +
       ROOT_DIR=/tmp/opam/
      -DEFAULT_INSTANCE="opam.ocaml.org"
      -INSTANCE=$1
      -INSTANCE_URL=$2
      -
      -if [ $INSTANCE = $DEFAULT_INSTANCE ]; then
      -  # Opam instance is considered the main instance so we must initialize the repository
      -  # if not present, otherwise we update it
      -  ( $CMD repo --all --root $ROOT_DIR | grep $INSTANCE_URL && $CMD update --root $ROOT_DIR ) \
      -      || $CMD init --reinit --bare --no-setup --root $ROOT_DIR $INSTANCE $INSTANCE_URL
      -else
      -  # Other instances, if present should be updated. If not present, they should be added
      -  # to the main root_dir as extra instances
      -  ( $CMD repo --all --root $ROOT_DIR | grep $INSTANCE_URL && $CMD update --root $ROOT_DIR ) \
      -    || $CMD repository add --root $ROOT_DIR $INSTANCE $INSTANCE_URL
      -fi
      +CMD=/usr/bin/opam
      +
      +function init_or_update_opam_root () {
      +  # Ensure opam root directory exists and references the main opam repository as default
      +  # instance. If not, create it from scratch. If properly configured, update the opam
      +  # repository
      +
      +  instance_name=$1
      +  instance_url=$2
      +
      +  if $CMD repo --all --root $ROOT_DIR | grep -qx $instance_url; then
      +    $CMD update --root $ROOT_DIR
      +  else
      +    $CMD init --reinit --bare --no-setup --root $ROOT_DIR $instance_name $instance_url
      +  fi
      +}
      +
      +function init_or_update_opam_instance () {
      +  # Ensure opam root directory exists and references the main opam repository as default
      +  # instance. If not, create it from scratch. If properly configured, update the opam
      +  # repository.
      +
      +  # Note that This assumes that the function init_or_update_opam_root already got called
      +  # prior to calling this one.
      +
      +  instance_name=$1
      +  instance_url=$2
      +
      +  if $CMD repo --all --root $ROOT_DIR | grep -qx $instance_url; then
      +    $CMD update --root $ROOT_DIR
      +  else
      +    $CMD repository add --root $ROOT_DIR --all-switches $instance_name $instance_url
      +  fi
      +}
      +
      +init_or_update_opam_root opam.ocaml.org https://opam.ocaml.org
      +
      +
      +init_or_update_opam_instance coq.inria.fr https://coq.inria.fr/opam/released
      +
      +init_or_update_opam_instance ocamlbench-repo https://github.com/OCamlPro/ocamlbench-repo.git
      +
*******************************************
  Service[opam-manage-shared-state-coq.inria.fr.timer] =>
   parameters =>
     enable =>
      - true
      + false
*******************************************
  Service[opam-manage-shared-state-ocamlbench-repo.timer] =>
   parameters =>
     enable =>
      - true
      + false
*******************************************
  Service[opam-manage-shared-state-opam.ocaml.org.timer] =>
   parameters =>
     enable =>
      - true
      + false
*******************************************
+ Service[opam-manage-shared-state.timer] =>
   parameters =>
      "enable": true
      "provider": "systemd"
*******************************************
  Systemd::Timer[opam-manage-shared-state-coq.inria.fr.timer] =>
   parameters =>
     enable =>
      - true
      + false
     ensure =>
      - present
      + absent
     service_content =>
      - # Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state (coq.inria.fr)
Requires=network.target
After=network.target

[Service]
User=swhworker
Group=swhworker
Type=simple
ExecStart=/usr/local/bin/opam-manage-shared-state.sh coq.inria.fr https://coq.inria.fr/opam/released

[Install]
WantedBy=multi-user.target

     timer_content =>
      - [Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-coq.inria.fr.service

[Unit]
Description=Software Heritage Manage OPAM shared state (coq.inria.fr) timer

*******************************************
  Systemd::Timer[opam-manage-shared-state-ocamlbench-repo.timer] =>
   parameters =>
     enable =>
      - true
      + false
     ensure =>
      - present
      + absent
     service_content =>
      - # Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state (ocamlbench-repo)
Requires=network.target
After=network.target

[Service]
User=swhworker
Group=swhworker
Type=simple
ExecStart=/usr/local/bin/opam-manage-shared-state.sh ocamlbench-repo https://github.com/OCamlPro/ocamlbench-repo.git

[Install]
WantedBy=multi-user.target

     timer_content =>
      - [Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-ocamlbench-repo.service

[Unit]
Description=Software Heritage Manage OPAM shared state (ocamlbench-repo) timer

*******************************************
  Systemd::Timer[opam-manage-shared-state-opam.ocaml.org.timer] =>
   parameters =>
     enable =>
      - true
      + false
     ensure =>
      - present
      + absent
     service_content =>
      - # Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state (opam.ocaml.org)
Requires=network.target
After=network.target

[Service]
User=swhworker
Group=swhworker
Type=simple
ExecStart=/usr/local/bin/opam-manage-shared-state.sh opam.ocaml.org https://opam.ocaml.org

[Install]
WantedBy=multi-user.target

     timer_content =>
      - [Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-opam.ocaml.org.service

[Unit]
Description=Software Heritage Manage OPAM shared state (opam.ocaml.org) timer

*******************************************
+ Systemd::Timer[opam-manage-shared-state.timer] =>
   parameters =>
      "enable": true
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "service_content": >>>
# Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state
Requires=network.target
Before=swh-worker@lister.service swh-worker@loader_opam.service
After=network.target

[Service]
User=swhworker
Group=swhworker
Type=oneshot
ExecStart=/usr/local/bin/opam-manage-shared-state.sh

[Install]
WantedBy=swh-worker@lister.service swh-worker@loader_opam.service
<<<
      "timer_content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
Persistent=true
OnCalendar=daily
RandomizedDelaySec=24h
FixedRandomDelay=true
Unit=.service

[Unit]
Description=Software Heritage Manage OPAM shared state timer
<<<
*******************************************
- Systemd::Unit_file[opam-manage-shared-state-coq.inria.fr.service]
*******************************************
  Systemd::Unit_file[opam-manage-shared-state-coq.inria.fr.timer] =>
   parameters =>
     content =>
      - [Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-coq.inria.fr.service

[Unit]
Description=Software Heritage Manage OPAM shared state (coq.inria.fr) timer

     enable =>
      - true
      + false
     ensure =>
      - present
      + absent
*******************************************
- Systemd::Unit_file[opam-manage-shared-state-ocamlbench-repo.service]
*******************************************
  Systemd::Unit_file[opam-manage-shared-state-ocamlbench-repo.timer] =>
   parameters =>
     content =>
      - [Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-ocamlbench-repo.service

[Unit]
Description=Software Heritage Manage OPAM shared state (ocamlbench-repo) timer

     enable =>
      - true
      + false
     ensure =>
      - present
      + absent
*******************************************
- Systemd::Unit_file[opam-manage-shared-state-opam.ocaml.org.service]
*******************************************
  Systemd::Unit_file[opam-manage-shared-state-opam.ocaml.org.timer] =>
   parameters =>
     content =>
      - [Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-opam.ocaml.org.service

[Unit]
Description=Software Heritage Manage OPAM shared state (opam.ocaml.org) timer

     enable =>
      - true
      + false
     ensure =>
      - present
      + absent
*******************************************
+ Systemd::Unit_file[opam-manage-shared-state.service] =>
   parameters =>
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
# Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state
Requires=network.target
Before=swh-worker@lister.service swh-worker@loader_opam.service
After=network.target

[Service]
User=swhworker
Group=swhworker
Type=oneshot
ExecStart=/usr/local/bin/opam-manage-shared-state.sh

[Install]
WantedBy=swh-worker@lister.service swh-worker@loader_opam.service
<<<
*******************************************
+ Systemd::Unit_file[opam-manage-shared-state.timer] =>
   parameters =>
      "enable": true
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
Persistent=true
OnCalendar=daily
RandomizedDelaySec=24h
FixedRandomDelay=true
Unit=.service

[Unit]
Description=Software Heritage Manage OPAM shared state timer
<<<
*******************************************
*** End octocatalog-diff on worker01.softwareheritage.org

puppet agent applied freshly (with this):

root@worker01:~# puppet agent --test
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for worker01.softwareheritage.org
Info: Applying configuration version '1641481571'
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/File[/usr/local/bin/opam-manage-shared-state.sh]/content:
--- /usr/local/bin/opam-manage-shared-state.sh  2022-01-06 15:08:38.499814335 +0000
+++ /tmp/puppet-file20220106-3113-1hqr2sm       2022-01-06 15:08:57.479814335 +0000
@@ -3,20 +3,51 @@
 # Managed by puppet class profile::swh::deploy::worker::loader_opam
 # Changes will be overwritten

-CMD=/usr/bin/opam
+# This script is in charge of bootstraping the opam root folder with the default opam
+# repository.
+
+set -e
+
 ROOT_DIR=/tmp/opam/
-DEFAULT_INSTANCE="opam.ocaml.org"
-INSTANCE=$1
-INSTANCE_URL=$2
-
-if [ $INSTANCE = $DEFAULT_INSTANCE ]; then
-  # Opam instance is considered the main instance so we must initialize the repository
-  # if not present, otherwise we update it
-  ( $CMD repo --all --root $ROOT_DIR | grep $INSTANCE_URL && $CMD update --root $ROOT_DIR ) \
-      || $CMD init --reinit --bare --no-setup --root $ROOT_DIR $INSTANCE $INSTANCE_URL
-else
-  # Other instances, if present should be updated. If not present, they should be added
-  # to the main root_dir as extra instances
-  ( $CMD repo --all --root $ROOT_DIR | grep $INSTANCE_URL && $CMD update --root $ROOT_DIR ) \
-    || $CMD repository add --root $ROOT_DIR $INSTANCE $INSTANCE_URL
-fi
+CMD=/usr/bin/opam
+
+function init_or_update_opam_root () {
+  # Ensure opam root directory exists and references the main opam repository as default
+  # instance. If not, create it from scratch. If properly configured, update the opam
+  # repository
+
+  instance_name=$1
+  instance_url=$2
+
+  if $CMD repo --all --root $ROOT_DIR | grep -qx $instance_url; then
+    $CMD update --root $ROOT_DIR
+  else
+    $CMD init --reinit --bare --no-setup --root $ROOT_DIR $instance_name $instance_url
+  fi
+}
+
+function init_or_update_opam_instance () {
+  # Ensure opam root directory exists and references the main opam repository as default
+  # instance. If not, create it from scratch. If properly configured, update the opam
+  # repository.
+
+  # Note that This assumes that the function init_or_update_opam_root already got called
+  # prior to calling this one.
+
+  instance_name=$1
+  instance_url=$2
+
+  if $CMD repo --all --root $ROOT_DIR | grep -qx $instance_url; then
+    $CMD update --root $ROOT_DIR
+  else
+    $CMD repository add --root $ROOT_DIR --all-switches $instance_name $instance_url
+  fi
+}
+
+init_or_update_opam_root opam.ocaml.org https://opam.ocaml.org
+
+
+init_or_update_opam_instance coq.inria.fr https://coq.inria.fr/opam/released
+
+init_or_update_opam_instance ocamlbench-repo https://github.com/OCamlPro/ocamlbench-repo.git
+

Info: Computing checksum on file /usr/local/bin/opam-manage-shared-state.sh
Info: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/File[/usr/local/bin/opam-manage-shared-state.sh]: Filebucketed /usr/local/bin/opam-manage-shared-state.sh to puppet with sum eddc370058408f8e14f31768f31d7d83
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/File[/usr/local/bin/opam-manage-shared-state.sh]/content: content changed '{md5}eddc370058408f8e14f31768f31d7d83' to '{md5}96e010bcc23be1fa7683c8255b73385e'
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-opam.ocaml.org.timer]/Systemd::Unit_file[opam-manage-shared-state-opam.ocaml.org.timer]/Service[opam-manage-shared-state-opam.ocaml.org.timer]/enable: enable changed 'true' to 'false'
Info: Computing checksum on file /etc/systemd/system/opam-manage-shared-state-opam.ocaml.org.timer
Info: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-opam.ocaml.org.timer]/Systemd::Unit_file[opam-manage-shared-state-opam.ocaml.org.timer]/File[/etc/systemd/system/opam-manage-shared-state-opam.ocaml.org.timer]: Filebucketed /etc/systemd/system/opam-manage-shared-state-opam.ocaml.org.timer to puppet with sum e34fc2fbabc002a55062d24c24ea8c90
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-opam.ocaml.org.timer]/Systemd::Unit_file[opam-manage-shared-state-opam.ocaml.org.timer]/File[/etc/systemd/system/opam-manage-shared-state-opam.ocaml.org.timer]/ensure: removed
Info: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-opam.ocaml.org.timer]/Systemd::Unit_file[opam-manage-shared-state-opam.ocaml.org.timer]/File[/etc/systemd/system/opam-manage-shared-state-opam.ocaml.org.timer]: Scheduling refresh of Class[Systemd::Systemctl::Daemon_reload]
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-coq.inria.fr.timer]/Systemd::Unit_file[opam-manage-shared-state-coq.inria.fr.timer]/Service[opam-manage-shared-state-coq.inria.fr.timer]/enable: enable changed 'true' to 'false'
Info: Computing checksum on file /etc/systemd/system/opam-manage-shared-state-coq.inria.fr.timer
Info: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-coq.inria.fr.timer]/Systemd::Unit_file[opam-manage-shared-state-coq.inria.fr.timer]/File[/etc/systemd/system/opam-manage-shared-state-coq.inria.fr.timer]: Filebucketed /etc/systemd/system/opam-manage-shared-state-coq.inria.fr.timer to puppet with sum c2db3f97df8328b934750754d5a20179
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-coq.inria.fr.timer]/Systemd::Unit_file[opam-manage-shared-state-coq.inria.fr.timer]/File[/etc/systemd/system/opam-manage-shared-state-coq.inria.fr.timer]/ensure: removed
Info: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-coq.inria.fr.timer]/Systemd::Unit_file[opam-manage-shared-state-coq.inria.fr.timer]/File[/etc/systemd/system/opam-manage-shared-state-coq.inria.fr.timer]: Scheduling refresh of Class[Systemd::Systemctl::Daemon_reload]
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-ocamlbench-repo.timer]/Systemd::Unit_file[opam-manage-shared-state-ocamlbench-repo.timer]/Service[opam-manage-shared-state-ocamlbench-repo.timer]/enable: enable changed 'true' to 'false'
Info: Computing checksum on file /etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.timer
Info: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-ocamlbench-repo.timer]/Systemd::Unit_file[opam-manage-shared-state-ocamlbench-repo.timer]/File[/etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.timer]: Filebucketed /etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.timer to puppet with sum 2dc9dd69c15b203c685a94c81f11a347
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-ocamlbench-repo.timer]/Systemd::Unit_file[opam-manage-shared-state-ocamlbench-repo.timer]/File[/etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.timer]/ensure: removed
Info: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-ocamlbench-repo.timer]/Systemd::Unit_file[opam-manage-shared-state-ocamlbench-repo.timer]/File[/etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.timer]: Scheduling refresh of Class[Systemd::Systemctl::Daemon_reload]
Notice: /Stage[main]/Profile::Systemd_journal::Journalbeat/Service[journalbeat]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Profile::Systemd_journal::Journalbeat/Service[journalbeat]: Unscheduling refresh on Service[journalbeat]
Info: Class[Systemd::Systemctl::Daemon_reload]: Scheduling refresh of Exec[systemctl-daemon-reload]
Notice: /Stage[main]/Systemd::Systemctl::Daemon_reload/Exec[systemctl-daemon-reload]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 3.29 seconds

Diff Detail

Repository

rSPSITE puppet-swh-site

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont requested review of this revision.Jan 6 2022, 11:49 AM

ardumont created this revision.

Harbormaster completed remote builds in B25830: Diff 24959.Jan 6 2022, 11:49 AM

Update

Herald added a reviewer: System administrators. · View Herald TranscriptJan 6 2022, 11:49 AM

Harbormaster completed remote builds in B25831: Diff 24960.Jan 6 2022, 11:50 AM

To avoid having to care about the order in which the instances are defined, I suggest making the script called by the timer unit itself hardcode the commands for the default instance, then use run-parts on the snippets generated for the other instances.

Thinking about this some more, as all the instances are defined by the same puppet class, we don't have to bother with run-parts at all, we could just generate a single script from a single template, which would have a couple of helper functions and generate function calls for every instance. Something like:

#!/bin/bash

set -e

ROOT_DIR=...
...

function init_or_update_opam_root () {
  ...
}

function init_or_update_opam_instance () {
  ...
}

init_or_update_opam_root <%= @default_instance_name %> <%= @default_instance_url %>

<% @other_instances.each do |name, url| %>
init_or_update_opam_instance <%= name %> <%= url %>
<% end %>

data/common/common.yaml
2556–2557	If we care about the order, it would be simpler to have a separate variable for the default instance.
site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state-default-instance.sh.erb
18–19 ↗	(On Diff #24960)	But this could probably just check whether $ROOT_DIR exists.
site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.timer.erb
5–6	This should be `RandomDelaySec`, to avoid all the timers firing at the same time. (probably with `FixedRandomDelay=true`).

ardumont edited the summary of this revision. (Show Details)Jan 6 2022, 2:58 PM

ardumont edited the test plan for this revision. (Show Details)

Adapt according to suggestions

Harbormaster completed remote builds in B25841: Diff 24970.Jan 6 2022, 2:58 PM

Fix to bash comments (and not "" ¯\_(ツ)_/¯)

Harbormaster completed remote builds in B25842: Diff 24971.Jan 6 2022, 3:01 PM

ardumont edited the test plan for this revision. (Show Details)Jan 6 2022, 3:01 PM

Actually make the script run ok. Prior to this update, the switch [1] stuff makes it
fail.

I've added the --all-switches so the command affects all switches (there will be only
one in our opam root directory).

Now everything is happy:

# journalctl -xef -u opam-manage-shared-state
Jan 06 14:10:16 worker01 opam-manage-shared-state.sh[4931]: [ERROR] Opam has not been initialised, please run `opam init'
Jan 06 14:10:16 worker01 opam-manage-shared-state.sh[4933]: [NOTE] Will configure from built-in defaults.
Jan 06 14:10:16 worker01 opam-manage-shared-state.sh[4933]: Checking for available remotes: rsync and local, git, mercurial, darcs. Perfect!
Jan 06 14:10:16 worker01 opam-manage-shared-state.sh[4933]: <><> Fetching repository information ><><><><><><><><><><><><><><><><><><><><><>
Jan 06 14:10:19 worker01 opam-manage-shared-state.sh[4933]: [opam.ocaml.org] Initialised
Jan 06 14:10:22 worker01 opam-manage-shared-state.sh[5027]: [coq.inria.fr] Initialised
Jan 06 14:10:25 worker01 opam-manage-shared-state.sh[5053]: [ocamlbench-repo] Initialised
Jan 06 14:10:25 worker01 systemd[1]: opam-manage-shared-state.service: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit opam-manage-shared-state.service has successfully entered the 'dead' state.
Jan 06 14:11:37 worker01 systemd[1]: Started Software Heritage Manage OPAM shared state.
░░ Subject: A start job for unit opam-manage-shared-state.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit opam-manage-shared-state.service has finished successfully.
░░
░░ The job identifier is 2043816.
Jan 06 14:11:37 worker01 opam-manage-shared-state.sh[7927]: [NOTE] Will configure from built-in defaults.
Jan 06 14:11:37 worker01 opam-manage-shared-state.sh[7927]: Checking for available remotes: rsync and local, git, mercurial, darcs. Perfect!
Jan 06 14:11:37 worker01 opam-manage-shared-state.sh[7927]: <><> Updating repositories ><><><><><><><><><><><><><><><><><><><><><><><><><><>
Jan 06 14:11:37 worker01 opam-manage-shared-state.sh[7927]: [coq.inria.fr] no changes from https://coq.inria.fr/opam/released
Jan 06 14:11:38 worker01 opam-manage-shared-state.sh[7927]: [ocamlbench-repo] no changes from git+https://github.com/OCamlPro/ocamlbench-repo.git
Jan 06 14:11:43 worker01 opam-manage-shared-state.sh[7927]: [opam.ocaml.org] no changes from https://opam.ocaml.org
Jan 06 14:11:47 worker01 opam-manage-shared-state.sh[8096]: [coq.inria.fr] no changes from https://coq.inria.fr/opam/released
Jan 06 14:11:48 worker01 opam-manage-shared-state.sh[8137]: [ocamlbench-repo] no changes from git+https://github.com/OCamlPro/ocamlbench-repo.git
Jan 06 14:11:48 worker01 systemd[1]: opam-manage-shared-state.service: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit opam-manage-shared-state.service has successfully entered the 'dead' state.

[1] opam is designed to hold any number of concurrent installation prefixes (see [2])

[2] https://opam.ocaml.org/doc/Manual.html#Switches

Harbormaster completed remote builds in B25844: Diff 24973.Jan 6 2022, 3:15 PM

Rework commit message with proper sentences

Harbormaster completed remote builds in B25846: Diff 24975.Jan 6 2022, 3:23 PM

ardumont edited the summary of this revision. (Show Details)Jan 6 2022, 3:23 PM

olasd added inline comments.Jan 6 2022, 3:30 PM

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.timer.erb
5–10	This doesn't execute the timer every four hours. What this does is: `OnCalendar=daily`: Try to execute the timer daily at midnight `AccuracySec=4h`: Execute the timer at the earliest point where the machine is being waken up for timer execution within an interval of 4 hours. In our always-on situation, it doesn't affect anything and could be kept at the default of 1 minute `RandomizedDelaySec=1200` and `FixedRandomDelay=true`: add a fixed random delay of up to 20 mins to the exact date of timer execution All of this means that timer will trigger every 24 hours, at a date within 20 minutes of midnight. If we want updates every 4 hours, we can use: OnCalendar=0..23/4:00:00 RandomizedDelaySec=4h FixedRandomDelay=true If we want updates every day, we can use: OnCalendar=daily RandomizedDelaySec=24h FixedRandomDelay=true

ardumont added inline comments.Jan 6 2022, 3:33 PM

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.timer.erb
5–10	d'oh right. I was focused on the remaining part of the diff. And then forget to double check that part. Thanks. I'm going with the last one (24h). That's frequent enough in regards to our current scheduling of the lister (1 day).

ardumont edited the summary of this revision. (Show Details)Jan 6 2022, 3:35 PM

Adapt timer period to 24h

Harbormaster completed remote builds in B25848: Diff 24977.Jan 6 2022, 3:36 PM

olasd added inline comments.Jan 6 2022, 3:46 PM

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.service.erb
1–19	These changes should make sure that the shared state is properly initialized before the opam lister and loader run. Maybe we should do that in a separate commit.

olasd added inline comments.Jan 6 2022, 3:51 PM

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.service.erb
1–19	Hmm, `RemainAfterExit=yes` will prevent the timer from activating the unit again (because the unit will remain active forever), so we should not add that. Now we should check if the `WantedBy`/`Before` are enough to make sure the `swh-worker@` units are happy to start, even if the unit ends up inactive (which will happen once the script ends). I'm also not sure this unit should be `WantedBy=multi-user.target` at all (which will activate the unit on boot, rather than only when required by other services).

olasd added inline comments.Jan 6 2022, 4:01 PM

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp
73–86	A bunch of these aren't being used. I think we need to drop the service unit too.

ardumont added inline comments.Jan 6 2022, 4:04 PM

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp
73–86	Oh yeah, it's still there because i did initally drop the services myself ^ (plus the next instruction). But that would fail to apply, saying it's already done by the next instruction.

Clean up unused variables

Harbormaster completed remote builds in B25850: Diff 24979.Jan 6 2022, 4:06 PM

ardumont edited the test plan for this revision. (Show Details)Jan 6 2022, 4:07 PM

olasd added inline comments.Jan 6 2022, 4:08 PM

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp
73–86	Ah, so the service removal is implicit when the timer is ensure => absent. Got it.

ardumont edited the test plan for this revision. (Show Details)Jan 6 2022, 4:10 PM

ardumont added inline comments.Jan 6 2022, 4:51 PM

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp
73–86	/me sighs it seems i'm wrong (test plan updated with the last puppet agent --test appliance shows that only the timer is dropped) fixing this and the other remarks you made

ardumont edited the test plan for this revision. (Show Details)Jan 6 2022, 5:00 PM

Adapt according to last suggestions

Harbormaster completed remote builds in B25853: Diff 24982.Jan 6 2022, 5:00 PM

ardumont added inline comments.Jan 6 2022, 5:03 PM

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp
73–86	ah no, ok. I know now, my first implementation was wong! Hence the mixed feeling here. I did a for-loop on each timer and service for each service. Hence the clash with the first instruction which already dropped the timer file.

Thanks for working on this!

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp
77	No need for the quoted variable here

This revision is now accepted and ready to land.Jan 7 2022, 3:45 AM

ardumont added inline comments.Jan 7 2022, 9:08 AM

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp
77	so true ;)

Drop unnecessary variable quoting

Harbormaster completed remote builds in B25858: Diff 24987.Jan 7 2022, 9:14 AM

Closed by commit rSPSITE791085c55b39: Make opam shared root initialization more robust (authored by ardumont). · Explain WhyJan 7 2022, 9:17 AM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rSPSITE791085c55b39: Make opam shared root initialization more robust.

Make opam shared root initialization more robust
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 24988

data/common/common.yaml

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.service.erb

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.sh.erb

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.timer.erb

Make opam shared root initialization more robustClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 24988

data/common/common.yaml

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.service.erb

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.sh.erb

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.timer.erb

Make opam shared root initialization more robust
ClosedPublic
Actions

Revision Contents
Changeset List