Details

Reviewers

ardumont
olasd

Group Reviewers

Reviewers

Maniphest Tasks

T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects

Commits

rDMODfc808e1fc9e5: model: Add 'id' field to RawExtrinsicMetadata

Summary

So that they can be properly deduplicated and referenced.

Depends on D4935.

Diff Detail

Repository

rDMOD Data model

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 19094
Build 29604: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 29603: arc lint + arc unit

Event Timeline

vlorentz created this revision.Jan 29 2021, 3:09 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJan 29 2021, 3:09 PM

Build is green

Patch application report for D4970 (id=17747)

Could not rebase; Attempt merge onto cad940dc8c...

Updating cad940d..0bcff7b
Fast-forward
 swh/model/hashutil.py               |   9 ++-
 swh/model/identifiers.py            |  79 ++++++++++++++++++++++++
 swh/model/model.py                  |   8 ++-
 swh/model/tests/test_identifiers.py | 117 ++++++++++++++++++++++++++++++++++++
 swh/model/tests/test_model.py       |   6 +-
 5 files changed, 216 insertions(+), 3 deletions(-)

Changes applied before test

commit 0bcff7b00bd30a7ee7af408bf00229c19785da55
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 29 15:08:49 2021 +0100

    model: Add 'id' field to RawExtrinsicMetadata
    
    So that they can be properly deduplicated and referenced.

commit 2a807789f22fdbf3838684b6b1ce8cf5a599b754
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jan 25 12:31:12 2021 +0100

    identifiers: Add raw_extrinsic_metadata_identifier
    
    This will be used to compute an intrisic identifier for RawExtrinsicMetadata;
    which can be used for deduplication and refering to it like any other sha1_git
    instead of needed to use a tuple of its fields.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/206/ for more details.

Harbormaster completed remote builds in B18888: Diff 17747.Jan 29 2021, 3:11 PM

vlorentz requested review of this revision.Jan 29 2021, 3:11 PM

lgtm

What's the impact for the production (migration script, code change, ...)?

Can you please reference the task this work is related to? T2703 i guess?

This revision is now accepted and ready to land.Feb 2 2021, 2:00 PM

It will start writing this field to kafka values, that's all

rebase

vlorentz added a child revision: D5004: identifiers: Change the manifest format of raw_extrinsic_metadata to use integer instead of ISO8601.Feb 4 2021, 11:06 AM

Build is green

Patch application report for D4970 (id=17867)

Rebasing onto 0c16581283...

Current branch diff-target is up to date.

Changes applied before test

commit 266b88dcaaa0cab48c67e62ebca51f0a4599c435
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 29 15:08:49 2021 +0100

    model: Add 'id' field to RawExtrinsicMetadata
    
    So that they can be properly deduplicated and referenced.

commit 272468f3b5a96c8854a26efe333c32cba4504aff
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jan 25 12:31:12 2021 +0100

    identifiers: Add raw_extrinsic_metadata_identifier
    
    This will be used to compute an intrisic identifier for RawExtrinsicMetadata;
    which can be used for deduplication and refering to it like any other sha1_git
    instead of needed to use a tuple of its fields.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/211/ for more details.

Harbormaster completed remote builds in B18989: Diff 17867.Feb 4 2021, 11:08 AM

vlorentz mentioned this in D5029: Add raw_extrinsic_metadata.id column in postgresql..Feb 5 2021, 2:42 PM

vlorentz mentioned this in D5030: raw_extrinsic_metadata: Make (target, authority_id, discovery_date, fetcher_id) non-unique.

fix base commit

Build is green

Patch application report for D4970 (id=17973)

Could not rebase; Attempt merge onto 0c16581283...

Updating 0c16581..266b88d
Fast-forward
 swh/model/hashutil.py               |   9 ++-
 swh/model/identifiers.py            |  79 ++++++++++++++++++++++++
 swh/model/model.py                  |   8 ++-
 swh/model/tests/test_identifiers.py | 117 ++++++++++++++++++++++++++++++++++++
 swh/model/tests/test_model.py       |   6 +-
 5 files changed, 216 insertions(+), 3 deletions(-)

Changes applied before test

commit 266b88dcaaa0cab48c67e62ebca51f0a4599c435
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 29 15:08:49 2021 +0100

    model: Add 'id' field to RawExtrinsicMetadata
    
    So that they can be properly deduplicated and referenced.

commit 272468f3b5a96c8854a26efe333c32cba4504aff
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jan 25 12:31:12 2021 +0100

    identifiers: Add raw_extrinsic_metadata_identifier
    
    This will be used to compute an intrisic identifier for RawExtrinsicMetadata;
    which can be used for deduplication and refering to it like any other sha1_git
    instead of needed to use a tuple of its fields.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/218/ for more details.

Harbormaster completed remote builds in B19094: Diff 17973.Feb 8 2021, 3:32 PM

big rebase, plz review again

vlorentz requested review of this revision.Mar 3 2021, 11:39 AM

Build is green

Patch application report for D4970 (id=18553)

Could not rebase; Attempt merge onto 8e0119962b...

Updating 8e01199..8cbe59e
Fast-forward
 swh/model/cli.py                    |  92 +++++---
 swh/model/hashutil.py               |   9 +-
 swh/model/identifiers.py            | 263 +++++++---------------
 swh/model/model.py                  |   8 +-
 swh/model/tests/test_cli.py         |   6 +-
 swh/model/tests/test_identifiers.py | 432 ++++++++++--------------------------
 swh/model/tests/test_model.py       |   7 +-
 7 files changed, 271 insertions(+), 546 deletions(-)

Changes applied before test

commit 8cbe59ee6251ac834a3d98b04146d2dfeb8da075
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 29 15:08:49 2021 +0100

    model: Add 'id' field to RawExtrinsicMetadata
    
    So that they can be properly deduplicated and referenced.

commit d88a5e13f2ffea5c0ebfad24e29ba41e86af20c0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jan 25 12:31:12 2021 +0100

    identifiers: Add raw_extrinsic_metadata_identifier
    
    This will be used to compute an intrisic identifier for RawExtrinsicMetadata;
    which can be used for deduplication and refering to it like any other sha1_git
    instead of needed to use a tuple of its fields.

commit bf4ab4336f7b43d442988c47d3dd70bb82b595c5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 3 10:44:48 2021 +0100

    identifiers: Remove the deprecated SWHID class
    
    Other packages don't use it anymore.

commit 1e924e84198a895003d6f649b8e3471cd93a7c7b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Mar 3 10:44:27 2021 +0100

    cli: stop using the deprecated SWHID class

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/274/ for more details.

Harbormaster completed remote builds in B19632: Diff 18553.Mar 3 2021, 11:42 AM

ardumont accepted this revision.Mar 4 2021, 11:06 AM

This revision is now accepted and ready to land.Mar 4 2021, 11:06 AM

olasd accepted this revision.Mar 4 2021, 11:06 AM

rebase, fix format of discovery_date

Build is green

Patch application report for D4970 (id=18587)

Could not rebase; Attempt merge onto bf4ab4336f...

Updating bf4ab43..fc808e1
Fast-forward
 swh/model/hashutil.py               |   9 ++-
 swh/model/identifiers.py            |  79 +++++++++++++++++++++++++
 swh/model/model.py                  |   8 ++-
 swh/model/tests/test_identifiers.py | 115 ++++++++++++++++++++++++++++++++++++
 swh/model/tests/test_model.py       |   7 ++-
 5 files changed, 215 insertions(+), 3 deletions(-)

Changes applied before test

commit fc808e1fc9e59ffae1e82ea483d529025d7d1436
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Jan 29 15:08:49 2021 +0100

    model: Add 'id' field to RawExtrinsicMetadata
    
    So that they can be properly deduplicated and referenced.

commit f6eab95253f13f28fe4d4652fc471e3e8a0b5565
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Jan 25 12:31:12 2021 +0100

    identifiers: Add raw_extrinsic_metadata_identifier
    
    This will be used to compute an intrisic identifier for RawExtrinsicMetadata;
    which can be used for deduplication and refering to it like any other sha1_git
    instead of needed to use a tuple of its fields.

See https://jenkins.softwareheritage.org/job/DMOD/job/tests-on-diff/278/ for more details.

Harbormaster completed remote builds in B19678: Diff 18587.Mar 4 2021, 11:23 AM

Closed by commit rDMODfc808e1fc9e5: model: Add 'id' field to RawExtrinsicMetadata (authored by vlorentz). · Explain WhyMar 4 2021, 3:10 PM

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDMODfc808e1fc9e5: model: Add 'id' field to RawExtrinsicMetadata.

model: Add 'id' field to RawExtrinsicMetadata
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Patch application report for D4970 (id=17747)

Changes applied before test

Patch application report for D4970 (id=17867)

Changes applied before test

Patch application report for D4970 (id=17973)

Changes applied before test

Patch application report for D4970 (id=18553)

Changes applied before test

Patch application report for D4970 (id=18587)

Changes applied before test

Revision Contents
Changeset List

Diff 17973

swh/model/identifiers.py

swh/model/model.py

swh/model/tests/test_identifiers.py

swh/model/tests/test_model.py

model: Add 'id' field to RawExtrinsicMetadataClosedPublicActions

Details

Diff Detail

Event Timeline

Patch application report for D4970 (id=17747)

Changes applied before test

Patch application report for D4970 (id=17867)

Changes applied before test

Patch application report for D4970 (id=17973)

Changes applied before test

Patch application report for D4970 (id=18553)

Changes applied before test

Patch application report for D4970 (id=18587)

Changes applied before test

Revision ContentsChangeset List

Diff 17973

swh/model/identifiers.py

swh/model/model.py

swh/model/tests/test_identifiers.py

swh/model/tests/test_model.py

model: Add 'id' field to RawExtrinsicMetadata
ClosedPublic
Actions

Revision Contents
Changeset List