Page MenuHomeSoftware Heritage
Differential D4807

Add an ExtID object
ClosedPublic
Actions

Authored by douardda on Jan 5 2021, 5:16 PM.

Details

Reviewers
vlorentz
olasd
Group Reviewers
Reviewers
Maniphest Tasks
T2849: Design and implement a mapping from "original VCS ids" to SWHIDs to help incremental loaders
Commits
rDMOD71be46174b87: Add an ExtID object
Summary

This needs tests

Related to T2849.

Diff Detail

Unit TestsFailed
View All

TimeTest
27 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.model.tests.test_hypothesis_strategies::test_dicts_generation_split_content
@given(object_dicts(split_content=True, blacklist_types=all_but_skipped_content)) > def test_dicts_generation_split_content(obj_type_and_obj):
50 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.model.tests.test_hypothesis_strategies::test_generation_split_content
@given(objects(split_content=True, blacklist_types=all_but_skipped_content)) > def test_generation_split_content(obj_type_and_obj):
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.model.merkle::swh.model.merkle.deep_update
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.model.tests.fields.test_compound.ValidateCompound::test_validate_against_schema
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.model.tests.fields.test_compound.ValidateCompound::test_validate_against_schema_field_error
View Full Test Results (2 Failed · 294 Passed · 2 Skipped)

Event Timeline

douardda created this revision.Jan 5 2021, 5:16 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJan 5 2021, 5:17 PM
swh-public-ci added a comment.Jan 5 2021, 5:18 PM

Build is green

Patch application report for D4807 (id=17038)

Rebasing onto 18fde50472...

Current branch diff-target is up to date.
Changes applied before test
commit 0366dc68e38b20933809142638db1cf73992cf2f
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Jan 4 12:33:04 2021 +0100

    Add an ExtID object

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/191/ for more details.

Harbormaster completed remote builds in B18179: Diff 17038.Jan 5 2021, 5:18 PM
douardda requested review of this revision.Jan 5 2021, 5:18 PM
douardda added a child revision: D4698: Add support for ExtID in the storage.Jan 5 2021, 5:21 PM
vlorentz added a subscriber: vlorentz.Jan 5 2021, 6:19 PM
vlorentz added inline comments.
swh/model/model.py
563

Not an enum?

vlorentz added inline comments.Jan 5 2021, 6:20 PM
swh/model/model.py
564–565

What about a SWHID instead?

douardda updated this revision to Diff 17091.Jan 8 2021, 3:08 PM

add a unique_key method to ExtID and some tests

swh-public-ci added a comment.Jan 8 2021, 3:09 PM

Build has FAILED

Patch application report for D4807 (id=17091)

Rebasing onto 18fde50472...

Current branch diff-target is up to date.
Changes applied before test
commit 0268627e43054049f8363ecb6497ee7516233416
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Jan 4 12:33:04 2021 +0100

    Add an ExtID object

Link to build: https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/192/
See console output for more information: https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/192/console

Harbormaster failed remote builds in B18231: Diff 17091!Jan 8 2021, 3:09 PM
douardda added inline comments.Jan 8 2021, 3:11 PM
swh/model/model.py
563

Not sure yet if we want an enum for this. So I started simple...

564–565

we do not (want) to support generic SWHID (with context), plus swhid is a textual representation that will need string manipulations / (en|de)coding, etc. so the idea was to stick, in the data model, to stricly what we need.

vlorentz added inline comments.Jan 8 2021, 3:36 PM
swh/model/model.py
564–565

we do not (want) to support generic SWHID (with context)

neither does raw_extrinsic_metadata, there's a validator to check it has no context.

douardda updated this revision to Diff 17095.Jan 8 2021, 3:41 PM

fix hypothesis strategies tests

swh-public-ci added a comment.Jan 8 2021, 3:42 PM

Build has FAILED

Patch application report for D4807 (id=17095)

Rebasing onto 18fde50472...

Current branch diff-target is up to date.
Changes applied before test
commit 0268627e43054049f8363ecb6497ee7516233416
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Jan 4 12:33:04 2021 +0100

    Add an ExtID object

Link to build: https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/193/
See console output for more information: https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/193/console

Harbormaster failed remote builds in B18238: Diff 17095!Jan 8 2021, 3:42 PM
douardda updated this revision to Diff 17096.Jan 8 2021, 3:44 PM

same but with all files commited

swh-public-ci added a comment.Jan 8 2021, 3:45 PM

Build is green

Patch application report for D4807 (id=17096)

Rebasing onto 18fde50472...

Current branch diff-target is up to date.
Changes applied before test
commit 10ade43f14328c09d931e0ce10df25f6e9231f6a
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Jan 4 12:33:04 2021 +0100

    Add an ExtID object

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/194/ for more details.

Harbormaster completed remote builds in B18239: Diff 17096.Jan 8 2021, 3:45 PM
douardda mentioned this in T3034: generalize usage of SWHID for referencing SWH archive objects.Feb 9 2021, 3:36 PM
douardda updated this revision to Diff 18625.Mar 5 2021, 11:26 AM

adapted to nes SWHID classes

swh-public-ci added a comment.Mar 5 2021, 11:27 AM

Build is green

Patch application report for D4807 (id=18625)

Rebasing onto 2185f9308e...

Current branch diff-target is up to date.
Changes applied before test
commit 1735c1a1f865c178b42991685b0c486cfd8a68c9
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Jan 4 12:33:04 2021 +0100

    Add an ExtID object
    
    this object aims at being able to keep in the SWH Archive an
      SWHID <-> External object ID
    map, e.g. to be able to keep track of Mercurial ids so the Mercurial
    loader can be made more efficient.
    
    Related to T2849.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/282/ for more details.

Harbormaster completed remote builds in B19718: Diff 18625.Mar 5 2021, 11:27 AM
douardda retitled this revision from [WIP] Add an ExtID object to Add an ExtID object.Mar 8 2021, 11:27 AM
douardda updated this revision to Diff 18686.Mar 8 2021, 4:24 PM

Rebase, make ExtID a HashableObject, and remove hypothesis strategies for ExtID

the mood is to get rid of those nowadays.

swh-public-ci added a comment.Mar 8 2021, 4:25 PM

Build is green

Patch application report for D4807 (id=18686)

Rebasing onto fca36585a3...

Current branch diff-target is up to date.
Changes applied before test
commit b2a0d79d08a6a4d531b84a3d034a40ffdb91cfa4
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Jan 4 12:33:04 2021 +0100

    Add an ExtID object
    
    this object aims at being able to keep in the SWH Archive an
      SWHID <-> External object ID
    map, e.g. to be able to keep track of Mercurial ids so the Mercurial
    loader can be made more efficient.
    
    Related to T2849.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/284/ for more details.

Harbormaster completed remote builds in B19779: Diff 18686.Mar 8 2021, 4:25 PM
vlorentz added inline comments.Mar 9 2021, 11:28 AM
swh/model/identifiers.py
833–834 ↗(On Diff #18686)

and should mention $StrWithoutSpaces is ASCII-encoded and something like this quote from the raw-ext-meta:

Newlines in $Bytes are escaped as with other git fields,
ie. by adding a space after them.
844 ↗(On Diff #18686)
846 ↗(On Diff #18686)

probably

swh/model/tests/swh_model_data.py
135–146 ↗(On Diff #18686)

;)

douardda updated this revision to Diff 18706.Mar 9 2021, 3:03 PM

Use CoreSWHID as target

swh-public-ci added a comment.Mar 9 2021, 3:04 PM

Build is green

Patch application report for D4807 (id=18706)

Rebasing onto fca36585a3...

Current branch diff-target is up to date.
Changes applied before test
commit c4ffa76fdf0ec974d446d0fe5347648d9cd5644c
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Jan 4 12:33:04 2021 +0100

    Add an ExtID object
    
    this object aims at being able to keep in the SWH Archive an
      SWHID <-> External object ID
    map, e.g. to be able to keep track of Mercurial ids so the Mercurial
    loader can be made more efficient.
    
    Related to T2849.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/285/ for more details.

Harbormaster completed remote builds in B19801: Diff 18706.Mar 9 2021, 3:04 PM
vlorentz accepted this revision.Mar 10 2021, 10:15 AM
This revision is now accepted and ready to land.Mar 10 2021, 10:15 AM
olasd accepted this revision.Mar 10 2021, 10:22 AM
olasd added a subscriber: olasd.

I wish we dropped the $object_identifier(dict) pattern in favor of a method of the model objects, but I guess it's consistent.

I wonder if the TEST_OBJECTS addition will break the tests of swh.storage until D4698 lands.

swh/model/model.py
1077–1083

You're manually handling all the attributes, why not just call the class initializer directly?

douardda added inline comments.Mar 10 2021, 10:29 AM
swh/model/model.py
1077–1083

possibly "just" the result of several refactorings... lemme check

douardda updated this revision to Diff 18720.Mar 10 2021, 10:40 AM

Apply vlorentz and olasd's suggestions

swh-public-ci added a comment.Mar 10 2021, 10:42 AM

Build is green

Patch application report for D4807 (id=18720)

Rebasing onto fca36585a3...

Current branch diff-target is up to date.
Changes applied before test
commit 71be46174b87d9658259517622fa689c68a003db
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Jan 4 12:33:04 2021 +0100

    Add an ExtID object
    
    this object aims at being able to keep in the SWH Archive an
      SWHID <-> External object ID
    map, e.g. to be able to keep track of Mercurial ids so the Mercurial
    loader can be made more efficient.
    
    Related to T2849.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/286/ for more details.

Harbormaster completed remote builds in B19815: Diff 18720.Mar 10 2021, 10:42 AM
douardda added a comment.Mar 10 2021, 3:55 PM
In D4807#132570, @olasd wrote:

I wonder if the TEST_OBJECTS addition will break the tests of swh.storage until D4698 lands.

yes it will break tests there...

Closed by commit rDMOD71be46174b87: Add an ExtID object (authored by douardda). · Explain WhyMar 11 2021, 2:17 PM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDMOD71be46174b87: Add an ExtID object.

Revision Contents
Changeset List

PathSize
swh/
model/
17 lines
24 lines
tests/
13 lines

Diff 17091

View Options

swh/model/hypothesis_strategies.py

Loading...
View Options

swh/model/model.py

Loading...
View Options

swh/model/tests/test_model.py

Loading...
About · Developer information · Legal