opam loader: Ensure required opam state is shared amongst ingestion/listing runs
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	ardumont
	Sep 20 2021, 2:51 PM

Description

P1170 shows that the opam state is around ~200Mib.
Potentially spawning this for the 3.3k origins does not sound reasonable.
As the disk is a precious resource currently in the workers (used amongst multiple ones).

So making sure this is reused amongst workers would be a plus.
That should not be a huge problem and can be dealt with at the infra level.

Revisions and Commits

rDLDBASE Generic VCS/Package Loader
		D6340	rDLDBASE8aca257ae8ec opam: Define a initialize_opam_root parameter for opam loader
		D6318	rDLDBASEb2d175965abe Allow opam loader to actually use multi-instance opam root
		D6317	rDLDBASEf4556e0113fa opam: Initialize opam root directory outside the constructor
rDLS Listers
	Abandoned		D6394 gitlab: Allow listing of instances providing multiple vcs_type [PLEASE DELETE ME]
		D6316	rDLSe7716c0122e9 opam: Share opam root directory even on multiple instances
		D6306	rDLS332ed8e54309 opam: Allow defining where to actually install the opam_root folder
		D6312	rDLSff5e86ff489e opam: Make the instance optional and derived from the url
rSPSITE puppet-swh-site
		D6305	rSPSITEf9dfffd41902 opam: Install and maintain up-to-date shared opam root directories

Related Objects
Search...

		Status	Assigned	Task
		Migrated	gitlab-migration	T3424 Opam support
		Migrated	gitlab-migration	T3590 opam loader: Ensure required opam state is shared amongst ingestion/listing runs

Event Timeline

ardumont triaged this task as Normal priority.Sep 20 2021, 2:51 PM

ardumont created this task.

ardumont added a revision: D6305: opam: Install and maintain up-to-date shared opam root directories.Sep 20 2021, 3:05 PM

ardumont added a revision: D6306: opam: Allow defining where to actually install the opam_root folder.Sep 20 2021, 3:10 PM

ardumont added a revision: D6312: opam: Make the instance optional and derived from the url.Sep 20 2021, 5:57 PM

ardumont mentioned this in rDLSb69b0b7fd674: opam: Move the state initialization into the get_pages method.Sep 21 2021, 11:23 AM

ardumont added a commit: rDLSff5e86ff489e: opam: Make the instance optional and derived from the url.

ardumont added a commit: rDLS332ed8e54309: opam: Allow defining where to actually install the opam_root folder.

ardumont renamed this task from opam loader: Ensure the current opam state is shared amongst ingestion to opam loader: Ensure required opam state is shared amongst ingestion/listing runs.Sep 21 2021, 7:28 PM

ardumont added a revision: D6317: opam: Initialize opam root directory outside the constructor.Sep 21 2021, 7:36 PM

ardumont added a commit: rDLDBASEf4556e0113fa: opam: Initialize opam root directory outside the constructor.Sep 22 2021, 12:18 PM

ardumont changed the task status from Open to Work in Progress.Sep 22 2021, 2:11 PM

ardumont claimed this task.

ardumont added a revision: D6318: opam: Allow shared state between loader runs using multi-instance opam root.

ardumont added a revision: D6316: opam: Share opam root directory even on multiple instances.

ardumont added a revision: D6340: opam: Define a initialize_opam_root parameter for opam loader.Sep 24 2021, 1:39 PM

ardumont added a commit: rDLSe7716c0122e9: opam: Share opam root directory even on multiple instances.Sep 29 2021, 9:15 AM

ardumont added a commit: rDLDBASEb2d175965abe: Allow opam loader to actually use multi-instance opam root.

ardumont added a commit: rDLDBASE8aca257ae8ec: opam: Define a initialize_opam_root parameter for opam loader.

ardumont added a commit: rSPSITEf9dfffd41902: opam: Install and maintain up-to-date shared opam root directories.Sep 29 2021, 9:35 AM

ardumont mentioned this in rSPSITE00613c49582d: loader_opam: Fix missing user and group definition.Sep 29 2021, 9:50 AM