Differential D6337

opam: Lock repository when executing maintenance on root directory
AbandonedPublicDraft
Actions

Authored by ardumont on Sep 24 2021, 12:06 PM.

Details

Reviewers

douardda

Group Reviewers

Reviewers

Summary

Is that the lock stanza you had in mind?

I have yet to exercize it though (in docker for example).

Another thing comes to mind, regarding the opam maintenance. I need to stop the loaders
from doing their own maintenance work (diff coming) because the lister should have done
it already.

I think it's fine for the lister to actually do that maintenance since that allows those
to run from another environment where no extra maintenance exists (e.g docker). For the
production part, there is D6305 which should be in charge of it (that currently installs
the opam root dir on all workers so they can do their job).

Test Plan

tox

Diff Detail

Repository

rDLS Listers

Branch

improve-multi-instance

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 23969
Build 37388: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 37387: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	3 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.opam.tests.test_lister::test_mock_init_repository_init mock_opam = (<MagicMock name='call' id='139966956782424'>, <MagicMock name='Popen' id='139966956782648'>) tmp_path = PosixPath('/tmp/pytest-of-jenkins/pytest-0/test_mock_init_repository_init0') datadir = '/var/lib/jenkins/workspace/DLS/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/lister/opam/tests/data'
	100 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.opam.tests.test_lister::test_opam_binary datadir = '/var/lib/jenkins/workspace/DLS/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/lister/opam/tests/data' swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f4c98c1e128>
	71 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.opam.tests.test_lister::test_opam_multi_instance datadir = '/var/lib/jenkins/workspace/DLS/tests-on-diff/.tox/py3/lib/python3.7/site-packages/swh/lister/opam/tests/data' swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f4c9a614cf8>
	81 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.bitbucket.tests.test_lister::test_bitbucket_full_lister
	1,277 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.bitbucket.tests.test_lister::test_bitbucket_incremental_lister
		View Full Test Results (3 Failed · 185 Passed)

Event Timeline

ardumont created this revision.Sep 24 2021, 12:06 PM

Build has FAILED

Patch application report for D6337 (id=23042)

Could not rebase; Attempt merge onto 5ab6b00408...

Updating 5ab6b00..4f84823
Fast-forward
 requirements-test.txt                |  1 +
 requirements.txt                     |  1 +
 swh/lister/opam/lister.py            | 78 +++++++++++++++++++++++++---------
 swh/lister/opam/tests/test_lister.py | 81 ++++++++++++++++++++++++++++++++----
 4 files changed, 135 insertions(+), 26 deletions(-)

Changes applied before test

commit 4f8482320b5a21e582eb1ecd71ba43b7dc637e84
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Sep 24 11:57:59 2021 +0200

    opam: Lock repository when executing maintenance on root directory

commit e7716c0122e9c4a411ed54dde821b48939eb2ebb
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Sep 21 18:18:49 2021 +0200

    opam: Share opam root directory even on multiple instances
    
    That avoids having multiple distinct opam root directories per opam lister instance. The
    current opam commands used by the lister are actually listing specifically per instance.
    
    Related to P1171

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/381/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/381/console

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 24 2021, 12:10 PM

Harbormaster failed remote builds in B23969: Diff 23042!

ardumont mentioned this in D6316: opam: Share opam root directory even on multiple instances.Sep 24 2021, 1:25 PM