Page MenuHomeSoftware Heritage

Make opam shared root initialization more robust
ClosedPublic

Authored by ardumont on Jan 6 2022, 11:49 AM.

Details

Summary

This:

  • drops the previous version of the tooling
  • installs the new tooling:
    • use set -e in the main script so command failures are immediately detected by icinga
    • the script triggered by the timer/service script updates all instances at once (in working order, first opam root with the default opam instance, then the other secondary repositories). Implementation wise, it's looping over all instances to init or update them all.
  • also changes the timer period to ~24h (this matches the scheduler configuration for

the opam lister)

Related to T3826

Test Plan
*******************************************
- File[/etc/systemd/system/opam-manage-shared-state-coq.inria.fr.service]
*******************************************
  File[/etc/systemd/system/opam-manage-shared-state-coq.inria.fr.timer] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
- File[/etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.service]
*******************************************
  File[/etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.timer] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
- File[/etc/systemd/system/opam-manage-shared-state-opam.ocaml.org.service]
*******************************************
  File[/etc/systemd/system/opam-manage-shared-state-opam.ocaml.org.timer] =>
   parameters =>
     ensure =>
      - file
      + absent
*******************************************
+ File[/etc/systemd/system/opam-manage-shared-state.service] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": "Class[Systemd::Systemctl::Daemon_reload]"
      "owner": "root"
      "show_diff": true
      "content": >>>
# Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state
Requires=network.target
Before=swh-worker@lister.service swh-worker@loader_opam.service
After=network.target

[Service]
User=swhworker
Group=swhworker
Type=oneshot
ExecStart=/usr/local/bin/opam-manage-shared-state.sh

[Install]
WantedBy=swh-worker@lister.service swh-worker@loader_opam.service
<<<
*******************************************
+ File[/etc/systemd/system/opam-manage-shared-state.timer] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": ["Class[Systemd::Systemctl::Daemon_reload]", "Service[opam-manage-shared-state.timer]"]
      "owner": "root"
      "show_diff": true
      "content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
Persistent=true
OnCalendar=daily
RandomizedDelaySec=24h
FixedRandomDelay=true
Unit=.service

[Unit]
Description=Software Heritage Manage OPAM shared state timer
<<<
*******************************************
+ File[/opam-manage-shared-state-coq.inria.fr.service] =>
   parameters =>
      "ensure": "absent"
*******************************************
+ File[/opam-manage-shared-state-ocamlbench-repo.service] =>
   parameters =>
      "ensure": "absent"
*******************************************
+ File[/opam-manage-shared-state-opam.ocaml.org.service] =>
   parameters =>
      "ensure": "absent"
*******************************************
  File[/usr/local/bin/opam-manage-shared-state.sh] =>
   parameters =>
     content =>
      @@ -4,19 +4,50 @@
       # Changes will be overwritten
      _
      -CMD=/usr/bin/opam
      +# This script is in charge of bootstraping the opam root folder with the default opam
      +# repository.
      +
      +set -e
      +
       ROOT_DIR=/tmp/opam/
      -DEFAULT_INSTANCE="opam.ocaml.org"
      -INSTANCE=$1
      -INSTANCE_URL=$2
      -
      -if [ $INSTANCE = $DEFAULT_INSTANCE ]; then
      -  # Opam instance is considered the main instance so we must initialize the repository
      -  # if not present, otherwise we update it
      -  ( $CMD repo --all --root $ROOT_DIR | grep $INSTANCE_URL && $CMD update --root $ROOT_DIR ) \
      -      || $CMD init --reinit --bare --no-setup --root $ROOT_DIR $INSTANCE $INSTANCE_URL
      -else
      -  # Other instances, if present should be updated. If not present, they should be added
      -  # to the main root_dir as extra instances
      -  ( $CMD repo --all --root $ROOT_DIR | grep $INSTANCE_URL && $CMD update --root $ROOT_DIR ) \
      -    || $CMD repository add --root $ROOT_DIR $INSTANCE $INSTANCE_URL
      -fi
      +CMD=/usr/bin/opam
      +
      +function init_or_update_opam_root () {
      +  # Ensure opam root directory exists and references the main opam repository as default
      +  # instance. If not, create it from scratch. If properly configured, update the opam
      +  # repository
      +
      +  instance_name=$1
      +  instance_url=$2
      +
      +  if $CMD repo --all --root $ROOT_DIR | grep -qx $instance_url; then
      +    $CMD update --root $ROOT_DIR
      +  else
      +    $CMD init --reinit --bare --no-setup --root $ROOT_DIR $instance_name $instance_url
      +  fi
      +}
      +
      +function init_or_update_opam_instance () {
      +  # Ensure opam root directory exists and references the main opam repository as default
      +  # instance. If not, create it from scratch. If properly configured, update the opam
      +  # repository.
      +
      +  # Note that This assumes that the function init_or_update_opam_root already got called
      +  # prior to calling this one.
      +
      +  instance_name=$1
      +  instance_url=$2
      +
      +  if $CMD repo --all --root $ROOT_DIR | grep -qx $instance_url; then
      +    $CMD update --root $ROOT_DIR
      +  else
      +    $CMD repository add --root $ROOT_DIR --all-switches $instance_name $instance_url
      +  fi
      +}
      +
      +init_or_update_opam_root opam.ocaml.org https://opam.ocaml.org
      +
      +
      +init_or_update_opam_instance coq.inria.fr https://coq.inria.fr/opam/released
      +
      +init_or_update_opam_instance ocamlbench-repo https://github.com/OCamlPro/ocamlbench-repo.git
      +
*******************************************
  Service[opam-manage-shared-state-coq.inria.fr.timer] =>
   parameters =>
     enable =>
      - true
      + false
*******************************************
  Service[opam-manage-shared-state-ocamlbench-repo.timer] =>
   parameters =>
     enable =>
      - true
      + false
*******************************************
  Service[opam-manage-shared-state-opam.ocaml.org.timer] =>
   parameters =>
     enable =>
      - true
      + false
*******************************************
+ Service[opam-manage-shared-state.timer] =>
   parameters =>
      "enable": true
      "provider": "systemd"
*******************************************
  Systemd::Timer[opam-manage-shared-state-coq.inria.fr.timer] =>
   parameters =>
     enable =>
      - true
      + false
     ensure =>
      - present
      + absent
     service_content =>
      - # Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state (coq.inria.fr)
Requires=network.target
After=network.target

[Service]
User=swhworker
Group=swhworker
Type=simple
ExecStart=/usr/local/bin/opam-manage-shared-state.sh coq.inria.fr https://coq.inria.fr/opam/released

[Install]
WantedBy=multi-user.target

     timer_content =>
      - [Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-coq.inria.fr.service

[Unit]
Description=Software Heritage Manage OPAM shared state (coq.inria.fr) timer

*******************************************
  Systemd::Timer[opam-manage-shared-state-ocamlbench-repo.timer] =>
   parameters =>
     enable =>
      - true
      + false
     ensure =>
      - present
      + absent
     service_content =>
      - # Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state (ocamlbench-repo)
Requires=network.target
After=network.target

[Service]
User=swhworker
Group=swhworker
Type=simple
ExecStart=/usr/local/bin/opam-manage-shared-state.sh ocamlbench-repo https://github.com/OCamlPro/ocamlbench-repo.git

[Install]
WantedBy=multi-user.target

     timer_content =>
      - [Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-ocamlbench-repo.service

[Unit]
Description=Software Heritage Manage OPAM shared state (ocamlbench-repo) timer

*******************************************
  Systemd::Timer[opam-manage-shared-state-opam.ocaml.org.timer] =>
   parameters =>
     enable =>
      - true
      + false
     ensure =>
      - present
      + absent
     service_content =>
      - # Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state (opam.ocaml.org)
Requires=network.target
After=network.target

[Service]
User=swhworker
Group=swhworker
Type=simple
ExecStart=/usr/local/bin/opam-manage-shared-state.sh opam.ocaml.org https://opam.ocaml.org

[Install]
WantedBy=multi-user.target

     timer_content =>
      - [Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-opam.ocaml.org.service

[Unit]
Description=Software Heritage Manage OPAM shared state (opam.ocaml.org) timer

*******************************************
+ Systemd::Timer[opam-manage-shared-state.timer] =>
   parameters =>
      "enable": true
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "service_content": >>>
# Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state
Requires=network.target
Before=swh-worker@lister.service swh-worker@loader_opam.service
After=network.target

[Service]
User=swhworker
Group=swhworker
Type=oneshot
ExecStart=/usr/local/bin/opam-manage-shared-state.sh

[Install]
WantedBy=swh-worker@lister.service swh-worker@loader_opam.service
<<<
      "timer_content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
Persistent=true
OnCalendar=daily
RandomizedDelaySec=24h
FixedRandomDelay=true
Unit=.service

[Unit]
Description=Software Heritage Manage OPAM shared state timer
<<<
*******************************************
- Systemd::Unit_file[opam-manage-shared-state-coq.inria.fr.service]
*******************************************
  Systemd::Unit_file[opam-manage-shared-state-coq.inria.fr.timer] =>
   parameters =>
     content =>
      - [Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-coq.inria.fr.service

[Unit]
Description=Software Heritage Manage OPAM shared state (coq.inria.fr) timer

     enable =>
      - true
      + false
     ensure =>
      - present
      + absent
*******************************************
- Systemd::Unit_file[opam-manage-shared-state-ocamlbench-repo.service]
*******************************************
  Systemd::Unit_file[opam-manage-shared-state-ocamlbench-repo.timer] =>
   parameters =>
     content =>
      - [Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-ocamlbench-repo.service

[Unit]
Description=Software Heritage Manage OPAM shared state (ocamlbench-repo) timer

     enable =>
      - true
      + false
     ensure =>
      - present
      + absent
*******************************************
- Systemd::Unit_file[opam-manage-shared-state-opam.ocaml.org.service]
*******************************************
  Systemd::Unit_file[opam-manage-shared-state-opam.ocaml.org.timer] =>
   parameters =>
     content =>
      - [Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-opam.ocaml.org.service

[Unit]
Description=Software Heritage Manage OPAM shared state (opam.ocaml.org) timer

     enable =>
      - true
      + false
     ensure =>
      - present
      + absent
*******************************************
+ Systemd::Unit_file[opam-manage-shared-state.service] =>
   parameters =>
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
# Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state
Requires=network.target
Before=swh-worker@lister.service swh-worker@loader_opam.service
After=network.target

[Service]
User=swhworker
Group=swhworker
Type=oneshot
ExecStart=/usr/local/bin/opam-manage-shared-state.sh

[Install]
WantedBy=swh-worker@lister.service swh-worker@loader_opam.service
<<<
*******************************************
+ Systemd::Unit_file[opam-manage-shared-state.timer] =>
   parameters =>
      "enable": true
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
Persistent=true
OnCalendar=daily
RandomizedDelaySec=24h
FixedRandomDelay=true
Unit=.service

[Unit]
Description=Software Heritage Manage OPAM shared state timer
<<<
*******************************************
*** End octocatalog-diff on worker01.softwareheritage.org

puppet agent applied freshly (with this):

root@worker01:~# puppet agent --test
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for worker01.softwareheritage.org
Info: Applying configuration version '1641481571'
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/File[/usr/local/bin/opam-manage-shared-state.sh]/content:
--- /usr/local/bin/opam-manage-shared-state.sh  2022-01-06 15:08:38.499814335 +0000
+++ /tmp/puppet-file20220106-3113-1hqr2sm       2022-01-06 15:08:57.479814335 +0000
@@ -3,20 +3,51 @@
 # Managed by puppet class profile::swh::deploy::worker::loader_opam
 # Changes will be overwritten

-CMD=/usr/bin/opam
+# This script is in charge of bootstraping the opam root folder with the default opam
+# repository.
+
+set -e
+
 ROOT_DIR=/tmp/opam/
-DEFAULT_INSTANCE="opam.ocaml.org"
-INSTANCE=$1
-INSTANCE_URL=$2
-
-if [ $INSTANCE = $DEFAULT_INSTANCE ]; then
-  # Opam instance is considered the main instance so we must initialize the repository
-  # if not present, otherwise we update it
-  ( $CMD repo --all --root $ROOT_DIR | grep $INSTANCE_URL && $CMD update --root $ROOT_DIR ) \
-      || $CMD init --reinit --bare --no-setup --root $ROOT_DIR $INSTANCE $INSTANCE_URL
-else
-  # Other instances, if present should be updated. If not present, they should be added
-  # to the main root_dir as extra instances
-  ( $CMD repo --all --root $ROOT_DIR | grep $INSTANCE_URL && $CMD update --root $ROOT_DIR ) \
-    || $CMD repository add --root $ROOT_DIR $INSTANCE $INSTANCE_URL
-fi
+CMD=/usr/bin/opam
+
+function init_or_update_opam_root () {
+  # Ensure opam root directory exists and references the main opam repository as default
+  # instance. If not, create it from scratch. If properly configured, update the opam
+  # repository
+
+  instance_name=$1
+  instance_url=$2
+
+  if $CMD repo --all --root $ROOT_DIR | grep -qx $instance_url; then
+    $CMD update --root $ROOT_DIR
+  else
+    $CMD init --reinit --bare --no-setup --root $ROOT_DIR $instance_name $instance_url
+  fi
+}
+
+function init_or_update_opam_instance () {
+  # Ensure opam root directory exists and references the main opam repository as default
+  # instance. If not, create it from scratch. If properly configured, update the opam
+  # repository.
+
+  # Note that This assumes that the function init_or_update_opam_root already got called
+  # prior to calling this one.
+
+  instance_name=$1
+  instance_url=$2
+
+  if $CMD repo --all --root $ROOT_DIR | grep -qx $instance_url; then
+    $CMD update --root $ROOT_DIR
+  else
+    $CMD repository add --root $ROOT_DIR --all-switches $instance_name $instance_url
+  fi
+}
+
+init_or_update_opam_root opam.ocaml.org https://opam.ocaml.org
+
+
+init_or_update_opam_instance coq.inria.fr https://coq.inria.fr/opam/released
+
+init_or_update_opam_instance ocamlbench-repo https://github.com/OCamlPro/ocamlbench-repo.git
+

Info: Computing checksum on file /usr/local/bin/opam-manage-shared-state.sh
Info: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/File[/usr/local/bin/opam-manage-shared-state.sh]: Filebucketed /usr/local/bin/opam-manage-shared-state.sh to puppet with sum eddc370058408f8e14f31768f31d7d83
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/File[/usr/local/bin/opam-manage-shared-state.sh]/content: content changed '{md5}eddc370058408f8e14f31768f31d7d83' to '{md5}96e010bcc23be1fa7683c8255b73385e'
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-opam.ocaml.org.timer]/Systemd::Unit_file[opam-manage-shared-state-opam.ocaml.org.timer]/Service[opam-manage-shared-state-opam.ocaml.org.timer]/enable: enable changed 'true' to 'false'
Info: Computing checksum on file /etc/systemd/system/opam-manage-shared-state-opam.ocaml.org.timer
Info: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-opam.ocaml.org.timer]/Systemd::Unit_file[opam-manage-shared-state-opam.ocaml.org.timer]/File[/etc/systemd/system/opam-manage-shared-state-opam.ocaml.org.timer]: Filebucketed /etc/systemd/system/opam-manage-shared-state-opam.ocaml.org.timer to puppet with sum e34fc2fbabc002a55062d24c24ea8c90
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-opam.ocaml.org.timer]/Systemd::Unit_file[opam-manage-shared-state-opam.ocaml.org.timer]/File[/etc/systemd/system/opam-manage-shared-state-opam.ocaml.org.timer]/ensure: removed
Info: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-opam.ocaml.org.timer]/Systemd::Unit_file[opam-manage-shared-state-opam.ocaml.org.timer]/File[/etc/systemd/system/opam-manage-shared-state-opam.ocaml.org.timer]: Scheduling refresh of Class[Systemd::Systemctl::Daemon_reload]
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-coq.inria.fr.timer]/Systemd::Unit_file[opam-manage-shared-state-coq.inria.fr.timer]/Service[opam-manage-shared-state-coq.inria.fr.timer]/enable: enable changed 'true' to 'false'
Info: Computing checksum on file /etc/systemd/system/opam-manage-shared-state-coq.inria.fr.timer
Info: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-coq.inria.fr.timer]/Systemd::Unit_file[opam-manage-shared-state-coq.inria.fr.timer]/File[/etc/systemd/system/opam-manage-shared-state-coq.inria.fr.timer]: Filebucketed /etc/systemd/system/opam-manage-shared-state-coq.inria.fr.timer to puppet with sum c2db3f97df8328b934750754d5a20179
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-coq.inria.fr.timer]/Systemd::Unit_file[opam-manage-shared-state-coq.inria.fr.timer]/File[/etc/systemd/system/opam-manage-shared-state-coq.inria.fr.timer]/ensure: removed
Info: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-coq.inria.fr.timer]/Systemd::Unit_file[opam-manage-shared-state-coq.inria.fr.timer]/File[/etc/systemd/system/opam-manage-shared-state-coq.inria.fr.timer]: Scheduling refresh of Class[Systemd::Systemctl::Daemon_reload]
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-ocamlbench-repo.timer]/Systemd::Unit_file[opam-manage-shared-state-ocamlbench-repo.timer]/Service[opam-manage-shared-state-ocamlbench-repo.timer]/enable: enable changed 'true' to 'false'
Info: Computing checksum on file /etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.timer
Info: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-ocamlbench-repo.timer]/Systemd::Unit_file[opam-manage-shared-state-ocamlbench-repo.timer]/File[/etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.timer]: Filebucketed /etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.timer to puppet with sum 2dc9dd69c15b203c685a94c81f11a347
Notice: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-ocamlbench-repo.timer]/Systemd::Unit_file[opam-manage-shared-state-ocamlbench-repo.timer]/File[/etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.timer]/ensure: removed
Info: /Stage[main]/Profile::Swh::Deploy::Worker::Loader_opam/Systemd::Timer[opam-manage-shared-state-ocamlbench-repo.timer]/Systemd::Unit_file[opam-manage-shared-state-ocamlbench-repo.timer]/File[/etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.timer]: Scheduling refresh of Class[Systemd::Systemctl::Daemon_reload]
Notice: /Stage[main]/Profile::Systemd_journal::Journalbeat/Service[journalbeat]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Profile::Systemd_journal::Journalbeat/Service[journalbeat]: Unscheduling refresh on Service[journalbeat]
Info: Class[Systemd::Systemctl::Daemon_reload]: Scheduling refresh of Exec[systemctl-daemon-reload]
Notice: /Stage[main]/Systemd::Systemctl::Daemon_reload/Exec[systemctl-daemon-reload]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 3.29 seconds

Diff Detail

Repository
rSPSITE puppet-swh-site
Branch
staging
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 25844
Build 40391: arc lint + arc unit

Event Timeline

ardumont created this revision.

To avoid having to care about the order in which the instances are defined, I suggest making the script called by the timer unit itself hardcode the commands for the default instance, then use run-parts on the snippets generated for the other instances.

Thinking about this some more, as all the instances are defined by the same puppet class, we don't have to bother with run-parts at all, we could just generate a single script from a single template, which would have a couple of helper functions and generate function calls for every instance. Something like:

#!/bin/bash

set -e

ROOT_DIR=...
...

function init_or_update_opam_root () {
  ...
}

function init_or_update_opam_instance () {
  ...
}

init_or_update_opam_root <%= @default_instance_name %> <%= @default_instance_url %>

<% @other_instances.each do |name, url| %>
init_or_update_opam_instance <%= name %> <%= url %>
<% end %>
data/common/common.yaml
2556–2557

If we care about the order, it would be simpler to have a separate variable for the default instance.

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state-default-instance.sh.erb
18–19 ↗(On Diff #24960)

But this could probably just check whether $ROOT_DIR exists.

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.timer.erb
5–6

This should be RandomDelaySec, to avoid all the timers firing at the same time. (probably with FixedRandomDelay=true).

ardumont edited the test plan for this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)
ardumont edited the test plan for this revision. (Show Details)

Adapt according to suggestions

Fix to bash comments (and not "" ¯\_(ツ)_/¯)

Actually make the script run ok. Prior to this update, the switch [1] stuff makes it
fail.

I've added the --all-switches so the command affects all switches (there will be only
one in our opam root directory).

Now everything is happy:

# journalctl -xef -u opam-manage-shared-state
Jan 06 14:10:16 worker01 opam-manage-shared-state.sh[4931]: [ERROR] Opam has not been initialised, please run `opam init'
Jan 06 14:10:16 worker01 opam-manage-shared-state.sh[4933]: [NOTE] Will configure from built-in defaults.
Jan 06 14:10:16 worker01 opam-manage-shared-state.sh[4933]: Checking for available remotes: rsync and local, git, mercurial, darcs. Perfect!
Jan 06 14:10:16 worker01 opam-manage-shared-state.sh[4933]: <><> Fetching repository information ><><><><><><><><><><><><><><><><><><><><><>
Jan 06 14:10:19 worker01 opam-manage-shared-state.sh[4933]: [opam.ocaml.org] Initialised
Jan 06 14:10:22 worker01 opam-manage-shared-state.sh[5027]: [coq.inria.fr] Initialised
Jan 06 14:10:25 worker01 opam-manage-shared-state.sh[5053]: [ocamlbench-repo] Initialised
Jan 06 14:10:25 worker01 systemd[1]: opam-manage-shared-state.service: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit opam-manage-shared-state.service has successfully entered the 'dead' state.
Jan 06 14:11:37 worker01 systemd[1]: Started Software Heritage Manage OPAM shared state.
░░ Subject: A start job for unit opam-manage-shared-state.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit opam-manage-shared-state.service has finished successfully.
░░
░░ The job identifier is 2043816.
Jan 06 14:11:37 worker01 opam-manage-shared-state.sh[7927]: [NOTE] Will configure from built-in defaults.
Jan 06 14:11:37 worker01 opam-manage-shared-state.sh[7927]: Checking for available remotes: rsync and local, git, mercurial, darcs. Perfect!
Jan 06 14:11:37 worker01 opam-manage-shared-state.sh[7927]: <><> Updating repositories ><><><><><><><><><><><><><><><><><><><><><><><><><><>
Jan 06 14:11:37 worker01 opam-manage-shared-state.sh[7927]: [coq.inria.fr] no changes from https://coq.inria.fr/opam/released
Jan 06 14:11:38 worker01 opam-manage-shared-state.sh[7927]: [ocamlbench-repo] no changes from git+https://github.com/OCamlPro/ocamlbench-repo.git
Jan 06 14:11:43 worker01 opam-manage-shared-state.sh[7927]: [opam.ocaml.org] no changes from https://opam.ocaml.org
Jan 06 14:11:47 worker01 opam-manage-shared-state.sh[8096]: [coq.inria.fr] no changes from https://coq.inria.fr/opam/released
Jan 06 14:11:48 worker01 opam-manage-shared-state.sh[8137]: [ocamlbench-repo] no changes from git+https://github.com/OCamlPro/ocamlbench-repo.git
Jan 06 14:11:48 worker01 systemd[1]: opam-manage-shared-state.service: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit opam-manage-shared-state.service has successfully entered the 'dead' state.

[1] opam is designed to hold any number of concurrent installation prefixes (see [2])

[2] https://opam.ocaml.org/doc/Manual.html#Switches

Rework commit message with proper sentences

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.timer.erb
5–10

This doesn't execute the timer every four hours.

What this does is:

  • OnCalendar=daily: Try to execute the timer daily at midnight
  • AccuracySec=4h: Execute the timer at the earliest point where the machine is being waken up for timer execution within an interval of 4 hours. In our always-on situation, it doesn't affect anything and could be kept at the default of 1 minute
  • RandomizedDelaySec=1200 and FixedRandomDelay=true: add a fixed random delay of up to 20 mins to the exact date of timer execution

All of this means that timer will trigger every 24 hours, at a date within 20 minutes of midnight.

If we want updates every 4 hours, we can use:

OnCalendar=0..23/4:00:00
RandomizedDelaySec=4h
FixedRandomDelay=true

If we want updates every day, we can use:

OnCalendar=daily
RandomizedDelaySec=24h
FixedRandomDelay=true
site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.timer.erb
5–10

d'oh right.
I was focused on the remaining part of the diff.
And then forget to double check that part.

Thanks.

I'm going with the last one (24h).
That's frequent enough in regards to our current scheduling of the lister (1 day).

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.service.erb
1–17

These changes should make sure that the shared state is properly initialized before the opam lister and loader run. Maybe we should do that in a separate commit.

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.service.erb
1–17

Hmm, RemainAfterExit=yes will prevent the timer from activating the unit again (because the unit will remain active forever), so we should not add that.

Now we should check if the WantedBy/Before are enough to make sure the swh-worker@ units are happy to start, even if the unit ends up inactive (which will happen once the script ends).

I'm also not sure this unit should be WantedBy=multi-user.target at all (which will activate the unit on boot, rather than only when required by other services).

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp
73–82

A bunch of these aren't being used. I think we need to drop the service unit too.

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp
73–82

Oh yeah, it's still there because i did initally drop the services myself ^ (plus the next instruction).
But that would fail to apply, saying it's already done by the next instruction.

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp
73–82

Ah, so the service removal is implicit when the timer is ensure => absent. Got it.

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp
73–82

/me sighs

it seems i'm wrong (test plan updated with the last puppet agent --test appliance shows that only the timer is dropped)

fixing this and the other remarks you made

Adapt according to last suggestions

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp
73–82

ah no, ok.
I know now, my first implementation was wong!
Hence the mixed feeling here.

I did a for-loop on each timer and service for each service.
Hence the clash with the first instruction which already dropped the timer file.

Thanks for working on this!

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp
77

No need for the quoted variable here

This revision is now accepted and ready to land.Jan 7 2022, 3:45 AM
site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp
77

so true ;)

Drop unnecessary variable quoting