Page MenuHomeSoftware Heritage
Differential D7985

[provenance] Implement a naive archive multiplexer
ClosedPublic
Actions

Authored by vsellier on Jun 14 2022, 12:23 PM.

Details

Reviewers
vlorentz
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T4315: [provenance] Fallback to swh-storage if a revision or its parent is not found in swh-graph
Commits
rDPROVb69c0f7689f0: Add a new multiplexed archive type
rDPROVf5ed9de87b39: Improve origin layer logs
rDPROVd45f066a8c51: Declare the missing swh-graph dependency
Summary
  • Improve origin layer logs
  • Add a missing dependency
  • Add a new multiplexed archive type

It queries all the underlying archives and returns
the first result found.
It a naive implementation but will allow to have
a fallback the the swh-storage if some is requested
in an outdated swh-graph

Related to T4315

Diff Detail

Repository
rDPROV Provenance database
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 29858
Build 46668: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 46667: arc lint + arc unit

Event Timeline

vsellier created this revision.Jun 14 2022, 12:23 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 14 2022, 12:23 PM
swh-public-ci added a comment.Jun 14 2022, 12:23 PM

Build has FAILED

Patch application report for D7985 (id=28772)

Rebasing onto d6e88dabb3...

Current branch diff-target is up to date.
Changes applied before test
commit 827318ce867b4041b5574e700d9e6de986c33674
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jun 14 12:15:54 2022 +0200

    Add a new multiplexed archive type
    
    It queries all the underlying archives and returns
    the first result found.
    It a naive implementation but will allow to have
    a fallback the the swh-storage if some is requested
    in an outdated swh-graph
    
    Related to T4315

commit f5ed9de87b392c62d59ad00e4f6f5a20a5954db6
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jun 14 12:21:10 2022 +0200

    Improve origin layer logs

commit d45f066a8c51677cd43f9ffafa25fef47df77368
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jun 14 12:18:37 2022 +0200

    Declare the missing swh-graph dependency

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/600/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/600/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 14 2022, 12:23 PM
Harbormaster failed remote builds in B29854: Diff 28772!
vsellier updated this revision to Diff 28775.Jun 14 2022, 2:37 PM

make the build happy

swh-public-ci added a comment.Jun 14 2022, 2:42 PM

Build has FAILED

Patch application report for D7985 (id=28775)

Rebasing onto d6e88dabb3...

Current branch diff-target is up to date.
Changes applied before test
commit ad4256e9fd38cac5aaf16c3c1e41759398cd51ab
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jun 14 12:15:54 2022 +0200

    Add a new multiplexed archive type
    
    It queries all the underlying archives and returns
    the first result found.
    It a naive implementation but will allow to have
    a fallback the the swh-storage if some is requested
    in an outdated swh-graph
    
    Related to T4315

commit f5ed9de87b392c62d59ad00e4f6f5a20a5954db6
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jun 14 12:21:10 2022 +0200

    Improve origin layer logs

commit d45f066a8c51677cd43f9ffafa25fef47df77368
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jun 14 12:18:37 2022 +0200

    Declare the missing swh-graph dependency

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/601/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/601/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jun 14 2022, 2:42 PM
Harbormaster failed remote builds in B29858: Diff 28775!
swh-public-ci added a comment.Jun 15 2022, 9:38 AM

Build is green

Patch application report for D7985 (id=28775)

Rebasing onto d6e88dabb3...

Current branch diff-target is up to date.
Changes applied before test
commit ad4256e9fd38cac5aaf16c3c1e41759398cd51ab
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jun 14 12:15:54 2022 +0200

    Add a new multiplexed archive type
    
    It queries all the underlying archives and returns
    the first result found.
    It a naive implementation but will allow to have
    a fallback the the swh-storage if some is requested
    in an outdated swh-graph
    
    Related to T4315

commit f5ed9de87b392c62d59ad00e4f6f5a20a5954db6
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jun 14 12:21:10 2022 +0200

    Improve origin layer logs

commit d45f066a8c51677cd43f9ffafa25fef47df77368
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jun 14 12:18:37 2022 +0200

    Declare the missing swh-graph dependency

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/602/ for more details.

Harbormaster completed remote builds in B29858: Diff 28775.Jun 15 2022, 9:38 AM
vsellier requested review of this revision.Jun 15 2022, 9:38 AM
vlorentz added a subscriber: vlorentz.Jun 15 2022, 9:45 AM

you forgot to select the base commit of the diff

vsellier added a comment.Jun 15 2022, 10:55 AM

I've deliberately created the diff with the 3 commits inside, I just forgot to update the title ;)

vsellier retitled this revision from Declare the missing swh-graph dependency to [provenance] Implement a naive archive multiplexer.Jun 15 2022, 10:57 AM
vsellier edited the summary of this revision. (Show Details)
vsellier added a project: Provenance database.
vlorentz accepted this revision.Jun 15 2022, 12:30 PM
vlorentz added inline comments.
swh/provenance/model.py
22–33

make sure we don't accidentally return an incorrect value

swh/provenance/multiplexer/archive.py
30

not related to this diff, but we really shouldn't use directory_ls. It's slow because it joins against other tables, but swh-provenance doesn't use any of the values obtained by the join. swh.storage.algos.directory.directory_get should be used instead.

This revision is now accepted and ready to land.Jun 15 2022, 12:30 PM
ardumont added a subscriber: ardumont.Jun 15 2022, 1:58 PM
ardumont added inline comments.
swh/provenance/tests/test_init.py
1
ardumont accepted this revision.Jun 15 2022, 1:59 PM

lgtm

(oops, forgot to validate it yesterday ;)

vsellier added inline comments.Jun 15 2022, 8:38 PM
swh/provenance/multiplexer/archive.py
30

Thanks for the suggestion, we need to define what must be improved / developed in provenance, I add this in the list.

vsellier updated this revision to Diff 28798.Jun 16 2022, 8:34 AM

Update according the reviews

  • Add and fix license headers
  • Ensure the _revisions_count variable was computed before returning its value
swh-public-ci added a comment.Jun 16 2022, 8:42 AM

Build is green

Patch application report for D7985 (id=28798)

Rebasing onto d6e88dabb3...

Current branch diff-target is up to date.
Changes applied before test
commit b69c0f7689f0d303eafc4f68c68a59ae418da4e9
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jun 14 12:15:54 2022 +0200

    Add a new multiplexed archive type
    
    It queries all the underlying archives and returns
    the first result found.
    It a naive implementation but will allow to have
    a fallback the the swh-storage if some is requested
    in an outdated swh-graph
    
    Related to T4315

commit f5ed9de87b392c62d59ad00e4f6f5a20a5954db6
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jun 14 12:21:10 2022 +0200

    Improve origin layer logs

commit d45f066a8c51677cd43f9ffafa25fef47df77368
Author: Vincent SELLIER <vincent.sellier@softwareheritage.org>
Date:   Tue Jun 14 12:18:37 2022 +0200

    Declare the missing swh-graph dependency

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/603/ for more details.

Harbormaster completed remote builds in B29880: Diff 28798.Jun 16 2022, 8:42 AM
Closed by commit rDPROVd45f066a8c51: Declare the missing swh-graph dependency (authored by vsellier). · Explain WhyJun 16 2022, 10:31 AM
This revision was automatically updated to reflect the committed changes.
vsellier added a commit: rDPROVd45f066a8c51: Declare the missing swh-graph dependency.
vsellier added a commit: rDPROVb69c0f7689f0: Add a new multiplexed archive type.
vsellier added a commit: rDPROVf5ed9de87b39: Improve origin layer logs.
olasd added a subscriber: olasd.Jun 17 2022, 3:42 PM
olasd added inline comments.
swh/provenance/multiplexer/archive.py
30

This code does not use the swh.storage implementations, but direct SQL queries implemented in swh.provenance.postgresql.archive.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
docker/
origin_client/
19 lines--
1 line--
swh/
provenance/
7 lines--
6 lines91%
Loading...
multiplexer/
78%
Loading...
77 lines91%
Loading...
24 lines98%
Loading...
tests/
34 lines97%
Loading...
28 lines100%
Loading...
tools/
origins/
6 lines--

Diff 28775

View Options

docker/origin_client/config.yml

Loading...
View Options

requirements-swh.txt

Loading...
View Options

swh/provenance/__init__.py

Loading...
View Options

swh/provenance/model.py

Loading...
View Options

swh/provenance/multiplexer/__init__.py

Loading...
View Options

swh/provenance/multiplexer/archive.py

Loading...
View Options

swh/provenance/origin.py

Loading...
View Options

swh/provenance/tests/test_archive_interface.py

Loading...
View Options

swh/provenance/tests/test_init.py

Loading...
View Options

swh/provenance/tools/origins/client.py

Loading...
About · Developer information · Legal