Paths

Table of Contentst

Differential D4346

package loaders: write extrinsic metadata to directories instead of revisions.
ClosedPublic
Actions

Authored by vlorentz on Oct 23 2020, 4:53 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases

Commits

rDLDBASEcf58604ccaa4: package loaders: write extrinsic metadata to directories instead of revisions.

Summary

They are more useful on directories, as directory ids are more intrinsic than
synthetic revision ids.
And this adds the revision swhid in the context, so the revision relationship
is still available when it's useful (eg. because the same directory can be
referenced from multiple revisions).

Note that package loaders still write the original_artifacts dict to the revision.

Part of the solution for T2668

Diff Detail

Repository

rDLDBASE Generic VCS/Package Loader

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 16710
Build 25767: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 25766: arc lint + arc unit

Event Timeline

vlorentz created this revision.Oct 23 2020, 4:53 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptOct 23 2020, 4:53 PM

vlorentz edited the summary of this revision. (Show Details)Oct 23 2020, 4:54 PM

Build is green

Patch application report for D4346 (id=15378)

Rebasing onto c30e485120...

Current branch diff-target is up to date.

Changes applied before test

commit 28bd28ca57a23d123dcc0931843e9a1d83c4d53d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Oct 23 16:52:23 2020 +0200

    package loaders: write extrinsic metadata to directories instead of revisions.
    
    They are more useful on directories, as directory ids are more intrinsic than
    synthetic revision ids.
    And this adds the revision swhid in the context, so the revision relationship
    is still available when it's useful (eg. because the same directory can be
    referenced from multiple revisions).

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/330/ for more details.

Harbormaster completed remote builds in B16605: Diff 15378.Oct 23 2020, 4:54 PM

vlorentz mentioned this in D4347: package loaders: write original_artifact metadata to directories instead of revisions..Oct 23 2020, 5:01 PM

vlorentz added a task: T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.Oct 23 2020, 5:01 PM

ardumont added a subscriber: moranegg.Oct 23 2020, 5:05 PM

vlorentz mentioned this in D4349: migrate_extrinsic_metadata: Write metadata on directories instead of revisions..Oct 23 2020, 5:26 PM

Couple of questions inline.

swh/loader/package/deposit/tests/test_deposit.py
215	Shouldn't this be `.DIRECTORY`? What did i misunderstand?
218	dir_meta?

moranegg added inline comments.Oct 28 2020, 2:59 PM

swh/loader/package/deposit/tests/test_deposit.py
218–223	dir_meta in each assertion when it is changed on line 220
403–404	is this variable supposed to stay rev_metadata?
406–407	rev_metadata => dir_metadata

fix names

rebase

Build is green

Patch application report for D4346 (id=15484)

Rebasing onto e2726ef650...

First, rewinding head to replay your work on top of it...
Applying: package loaders: write extrinsic metadata to directories instead of revisions.

Changes applied before test

commit 48e60d7d3b8c1baa78cb4252d9005b672237aa57
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Oct 23 16:52:23 2020 +0200

    package loaders: write extrinsic metadata to directories instead of revisions.
    
    They are more useful on directories, as directory ids are more intrinsic than
    synthetic revision ids.
    And this adds the revision swhid in the context, so the revision relationship
    is still available when it's useful (eg. because the same directory can be
    referenced from multiple revisions).

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/333/ for more details.

Harbormaster completed remote builds in B16710: Diff 15484.Oct 29 2020, 11:10 AM

ardumont added inline comments.Oct 29 2020, 11:11 AM

swh/loader/package/deposit/tests/test_deposit.py
215	why did the test passed with the wong `.REVISION`?

Build is green

Patch application report for D4346 (id=15486)

Rebasing onto e2726ef650...

Current branch diff-target is up to date.

Changes applied before test

commit cf58604ccaa469a7d31c30c476a3b9187493e9ce
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Oct 23 16:52:23 2020 +0200

    package loaders: write extrinsic metadata to directories instead of revisions.
    
    They are more useful on directories, as directory ids are more intrinsic than
    synthetic revision ids.
    And this adds the revision swhid in the context, so the revision relationship
    is still available when it's useful (eg. because the same directory can be
    referenced from multiple revisions).

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/335/ for more details.

Harbormaster completed remote builds in B16712: Diff 15486.Oct 29 2020, 11:13 AM

vlorentz marked 6 inline comments as done.Oct 29 2020, 11:19 AM

vlorentz added inline comments.

swh/loader/package/deposit/tests/test_deposit.py
215	because the storage only checks if it's `MetadataTargetType.ORIGIN` or not. We should probably add a check to make sure it matches the type in the SWHID