Details

Reviewers

vsellier

Group Reviewers

System administrators

Maniphest Tasks

T3590: opam loader: Ensure required opam state is shared amongst ingestion/listing runs

Commits

rSPSITEf9dfffd41902: opam: Install and maintain up-to-date shared opam root directories

Summary

(on workers for opam lister and loader services)

This shall reduce the disk use to one opam root per worker (instead of worst case
scenario of x distincts opam root * y distincts instances, with x the number of workers,
y the unknown number of distinct opam instances).

Using /tmp/opam as default sounds sensible enough (providing we make sure
that it's initialized at boot time).

(then other diffs are required on the lister/loader for this to actually work [1] [2] [3].
They are adaptations for the lister and loader to actually use correctcly such opam root).

[1] D6306
[2] D6316 (lister)
[3] D6318 (loader)

Related to T3590

Test Plan

octo-diff on worker staging node (opam loader deployed only there for now):

+ File[/etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.service] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": "Class[Systemd::Systemctl::Daemon_reload]"
      "owner": "root"
      "show_diff": true
      "content": >>>
# Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state (ocamlbench-repo)
Requires=network.target
After=network.target

[Service]
User=
Group=
Type=simple
ExecStart=( /usr/bin/opam repo --all --root /tmp/opam/ | grep https://github.com/OCamlPro/ocamlbench-repo.git && /usr/bin/opam update --root /tmp/opam/ ) || /usr/bin/opam repository add --root /tmp/opam/ ocamlbench-repo https://github.com/OCamlPro/ocamlbench-repo.git

[Install]
WantedBy=multi-user.target
<<<
*******************************************
+ File[/etc/systemd/system/opam-manage-shared-state-ocamlbench-repo.timer] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": ["Class[Systemd::Systemctl::Daemon_reload]", "Service[opam-manage-shared-state-ocamlbench-repo.timer]"]
      "owner": "root"
      "show_diff": true
      "content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-ocamlbench-repo.service

[Unit]
Description=Software Heritage Manage OPAM shared state (ocamlbench-repo) timer
<<<
*******************************************
+ File[/etc/systemd/system/opam-manage-shared-state-opam.service] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": "Class[Systemd::Systemctl::Daemon_reload]"
      "owner": "root"
      "show_diff": true
      "content": >>>
# Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state (opam)
Requires=network.target
After=network.target

[Service]
User=
Group=
Type=simple
ExecStart=( /usr/bin/opam repo --all --root /tmp/opam/ | grep https://opam.ocaml.org && /usr/bin/opam update --root /tmp/opam/ ) || /usr/bin/opam init --reinit --bare --no-setup --root /tmp/opam/ opam https://opam.ocaml.org

[Install]
WantedBy=multi-user.target
<<<
*******************************************
+ File[/etc/systemd/system/opam-manage-shared-state-opam.timer] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": ["Class[Systemd::Systemctl::Daemon_reload]", "Service[opam-manage-shared-state-opam.timer]"]
      "owner": "root"
      "show_diff": true
      "content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-opam.service

[Unit]
Description=Software Heritage Manage OPAM shared state (opam) timer
<<<
*******************************************
+ File[/tmp/opam] =>
   parameters =>
      "ensure": "directory"
      "mode": "0644"
      "recurse": true
*******************************************
+ Service[opam-manage-shared-state-ocamlbench-repo.timer] =>
   parameters =>
      "enable": true
      "provider": "systemd"
*******************************************
+ Service[opam-manage-shared-state-opam.timer] =>
   parameters =>
      "enable": true
      "provider": "systemd"
*******************************************
+ Systemd::Timer[opam-manage-shared-state-ocamlbench-repo.timer] =>
   parameters =>
      "enable": true
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "service_content": >>>
# Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state (ocamlbench-repo)
Requires=network.target
After=network.target

[Service]
User=
Group=
Type=simple
ExecStart=( /usr/bin/opam repo --all --root /tmp/opam/ | grep https://github.com/OCamlPro/ocamlbench-repo.git && /usr/bin/opam update --root /tmp/opam/ ) || /usr/bin/opam repository add --root /tmp/opam/ ocamlbench-repo https://github.com/OCamlPro/ocamlbench-repo.git

[Install]
WantedBy=multi-user.target
<<<
      "timer_content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-ocamlbench-repo.service

[Unit]
Description=Software Heritage Manage OPAM shared state (ocamlbench-repo) timer
<<<
*******************************************
+ Systemd::Timer[opam-manage-shared-state-opam.timer] =>
   parameters =>
      "enable": true
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "service_content": >>>
# Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state (opam)
Requires=network.target
After=network.target

[Service]
User=
Group=
Type=simple
ExecStart=( /usr/bin/opam repo --all --root /tmp/opam/ | grep https://opam.ocaml.org && /usr/bin/opam update --root /tmp/opam/ ) || /usr/bin/opam init --reinit --bare --no-setup --root /tmp/opam/ opam https://opam.ocaml.org

[Install]
WantedBy=multi-user.target
<<<
      "timer_content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-opam.service

[Unit]
Description=Software Heritage Manage OPAM shared state (opam) timer
<<<
*******************************************
+ Systemd::Unit_file[opam-manage-shared-state-ocamlbench-repo.service] =>
   parameters =>
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
# Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state (ocamlbench-repo)
Requires=network.target
After=network.target

[Service]
User=
Group=
Type=simple
ExecStart=( /usr/bin/opam repo --all --root /tmp/opam/ | grep https://github.com/OCamlPro/ocamlbench-repo.git && /usr/bin/opam update --root /tmp/opam/ ) || /usr/bin/opam repository add --root /tmp/opam/ ocamlbench-repo https://github.com/OCamlPro/ocamlbench-repo.git

[Install]
WantedBy=multi-user.target
<<<
*******************************************
+ Systemd::Unit_file[opam-manage-shared-state-ocamlbench-repo.timer] =>
   parameters =>
      "enable": true
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-ocamlbench-repo.service

[Unit]
Description=Software Heritage Manage OPAM shared state (ocamlbench-repo) timer
<<<
*******************************************
+ Systemd::Unit_file[opam-manage-shared-state-opam.service] =>
   parameters =>
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
# Opam manage shared state unit file
# Managed by puppet class profile::swh::deploy::worker::loader_opam
# Changes will be overwritten

[Unit]
Description=Software Heritage Manage OPAM shared state (opam)
Requires=network.target
After=network.target

[Service]
User=
Group=
Type=simple
ExecStart=( /usr/bin/opam repo --all --root /tmp/opam/ | grep https://opam.ocaml.org && /usr/bin/opam update --root /tmp/opam/ ) || /usr/bin/opam init --reinit --bare --no-setup --root /tmp/opam/ opam https://opam.ocaml.org

[Install]
WantedBy=multi-user.target
<<<
*******************************************
+ Systemd::Unit_file[opam-manage-shared-state-opam.timer] =>
   parameters =>
      "enable": true
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
AccuracySec=4h
Persistent=true
Unit=opam-manage-shared-state-opam.service

[Unit]
Description=Software Heritage Manage OPAM shared state (opam) timer
<<<
*******************************************
*** End octocatalog-diff on worker0.internal.staging.swh.network

Here are the commands that are actually executed by the service for each instance. They
are running as expected (from a scratch opam root directory):

1. no opam instance then install it
2. if opam instance already installed, just update it
3. if ocamlbench-pro not present, just add it to the same opam root directory
4. if ocamlbench-pro present, just update it):

$ rm -rf /tmp/opam
$ ( /usr/bin/opam repo --all --root /tmp/opam/ | grep https://opam.ocaml.org && /usr/bin/opam update --root /tmp/opam/ ) || /usr/bin/opam init --reinit --bare --no-setup --root /tmp/opam/ opam https://opam.ocaml.org
[ERROR] Opam has not been initialised, please run `opam init'
[NOTE] Will configure from built-in defaults.
Checking for available remotes: rsync and local, git, mercurial, darcs. Perfect!

<><> Fetching repository information ><><><><><><><><><><><><><><><><><><><><><>
[opam] Initialised
$ ( /usr/bin/opam repo --all --root /tmp/opam/ | grep https://opam.ocaml.org && /usr/bin/opam update --root /tmp/opam/ ) || /usr/bin/opam init --reinit --bare --no-setup --root /tmp/opam/ opam https://opam.ocaml.org
opam         https://opam.ocaml.org <default>(1)

<><> Updating package repositories ><><><><><><><><><><><><><><><><><><><><><><>
[opam] no changes from https://opam.ocaml.org
$ ( /usr/bin/opam repo --all --root /tmp/opam/ | grep https://github.com/OCamlPro/ocamlbench-repo.git && /usr/bin/opam update --root /tmp/opam/ ) || /usr/bin/opam repository add --root /tmp/opam/ ocamlbench-repo https://github.com/OCamlPro/ocamlbench-repo.git
[WARNING] No switch is currently set, perhaps you meant '--set-default'?
[ocamlbench-repo] Initialised
[ERROR] No switch is currently set. Please use 'opam switch' to set or install a switch
$ ( /usr/bin/opam repo --all --root /tmp/opam/ | grep https://github.com/OCamlPro/ocamlbench-repo.git && /usr/bin/opam update --root /tmp/opam/ ) || /usr/bin/opam repository add --root /tmp/opam/ ocamlbench-repo https://github.com/OCamlPro/ocamlbench-repo.git
ocamlbench-repo git+https://github.com/OCamlPro/ocamlbench-repo.git

<><> Updating package repositories ><><><><><><><><><><><><><><><><><><><><><><>
[opam] no changes from https://opam.ocaml.org
$ /usr/bin/opam repo --all --root /tmp/opam/
# Repository    # Url                                               # Switches(rank)
ocamlbench-repo git+https://github.com/OCamlPro/ocamlbench-repo.git
opam            https://opam.ocaml.org                              <default>(1)

Diff Detail

Repository

rSPSITE puppet-swh-site

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont requested review of this revision.Sep 20 2021, 3:05 PM

ardumont created this revision.

Harbormaster completed remote builds in B23820: Diff 22894.Sep 20 2021, 3:05 PM

ardumont mentioned this in D6306: opam: Allow defining where to actually install the opam_root folder.Sep 20 2021, 3:10 PM

ardumont edited the summary of this revision. (Show Details)Sep 20 2021, 3:11 PM

ardumont edited the summary of this revision. (Show Details)

Given the D6306 rework, this requires a bit of change (but nothing too drastic).

ardumont added inline comments.Sep 21 2021, 1:16 PM

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.timer.erb
6	oops!

Fix typo on the timer template

Harbormaster completed remote builds in B23863: Diff 22934.Sep 21 2021, 1:41 PM

ardumont planned changes to this revision.Sep 21 2021, 1:41 PM

ardumont mentioned this in D6316: opam: Share opam root directory even on multiple instances.Sep 21 2021, 7:26 PM

ardumont retitled this revision from opam: Actually install shared state amongst worker to opam: Install and maintain up-to-date shared opam root directories.Sep 22 2021, 7:02 PM

ardumont edited the summary of this revision. (Show Details)

ardumont edited the test plan for this revision. (Show Details)

Update

Harbormaster completed remote builds in B23912: Diff 22986.Sep 22 2021, 7:04 PM

ardumont edited the summary of this revision. (Show Details)Sep 22 2021, 7:04 PM

ardumont edited the test plan for this revision. (Show Details)

Refactor and make the install/update idempotent

Harbormaster completed remote builds in B23914: Diff 22988.Sep 22 2021, 7:19 PM

ardumont edited the test plan for this revision. (Show Details)Sep 22 2021, 7:23 PM

ardumont added inline comments.Sep 23 2021, 9:29 AM

data/common/common.yaml
2537	Added another opam instance for demonstration (i guess we can keep it as well, when it's time to deploy, i'll add both repositories for crawling).

ardumont mentioned this in D6337: opam: Lock repository when executing maintenance on root directory.Sep 24 2021, 12:06 PM

ardumont mentioned this in D6340: opam: Define a initialize_opam_root parameter for opam loader.Sep 24 2021, 1:48 PM

ardumont edited the summary of this revision. (Show Details)Sep 24 2021, 2:12 PM

It looks ok for the puppet code

This revision is now accepted and ready to land.Sep 24 2021, 4:15 PM

Rebase

Harbormaster completed remote builds in B24069: Diff 23155.Sep 29 2021, 9:05 AM

Closed by commit rSPSITEf9dfffd41902: opam: Install and maintain up-to-date shared opam root directories (authored by ardumont). · Explain WhySep 29 2021, 9:35 AM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rSPSITEf9dfffd41902: opam: Install and maintain up-to-date shared opam root directories.

opam: Install and maintain up-to-date shared opam root directories
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 23160

data/common/common.yaml

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.service.erb

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.timer.erb

opam: Install and maintain up-to-date shared opam root directoriesClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 23160

data/common/common.yaml

site-modules/profile/manifests/swh/deploy/worker/loader_opam.pp

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.service.erb

site-modules/profile/templates/swh/deploy/loader_opam/opam-manage-shared-state.timer.erb

opam: Install and maintain up-to-date shared opam root directories
ClosedPublic
Actions

Revision Contents
Changeset List