Page MenuHomeSoftware Heritage
Differential D8656

Normalize _add() methods of the ProvenanceStorage interface
ClosedPublic
Actions

Authored by douardda on Oct 11 2022, 12:24 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T4616: Add kafka journal log for the revision layer
Commits
rDPROV7e6a62c990b7: Rename ProvenanceInterface.directory_xxx_flattenned as directory_xxx_flattened
rDPROV2bd74fc7d97d: Normalize _add() methods of the ProvenanceStorage interface
Summary

make them all accept a Dict[Sha1Git, xxx] as argument, ie:

  • remove support for Iterable[bytes] in revision_add, and
  • replace Iterable[bytes] by Dict[Sha1Git, bytes] for location_add

Currently, the sha1 of location path in location_add() is not really
used by any backend, so the computation of said hashed is a waste of
resource, but it makes the API of this interface much more consistent
which will be helpful for coming features (like kafka journal).

Related to T4616

Diff Detail

Repository
rDPROV Provenance database
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Oct 11 2022, 12:24 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 11 2022, 12:24 PM
douardda added a child revision: D8657: Add support for kafka journalization of the ProvenanceStorageInterface.Oct 11 2022, 12:25 PM
swh-public-ci added a comment.Oct 11 2022, 12:31 PM

Build has FAILED

Patch application report for D8656 (id=31244)

Rebasing onto 6f4a193e90...

Current branch diff-target is up to date.
Changes applied before test
commit d182fcdbc532f1769bec129b2e37c36a59d021de
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 7 14:51:09 2022 +0200

    Normalize _add() methods of the ProvenanceStorage interface
    
    make them all accept a Dict[Sha1Git, xxx] as argument, ie:
    
    - remove support for Iterable[bytes] in revision_add, and
    - replace Iterable[bytes] by Dict[Sha1Git, bytes] for location_add
    
    Currently, the sha1 of location path in location_add() is not really
    used by any backend, so the computation of said hashed is a waste of
    resource, but it makes the API of this interface much more consistent
    which will be helpful for coming features (like kafka journal).

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/673/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/673/console

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 11 2022, 12:31 PM
Harbormaster failed remote builds in B32196: Diff 31244!
douardda updated this revision to Diff 31249.Oct 11 2022, 2:01 PM

typo

swh-public-ci added a comment.Oct 11 2022, 2:07 PM

Build has FAILED

Patch application report for D8656 (id=31249)

Rebasing onto 6f4a193e90...

Current branch diff-target is up to date.
Changes applied before test
commit 6b539ecbe1673caf539e0636de76881a3f8ed171
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 7 14:51:09 2022 +0200

    Normalize _add() methods of the ProvenanceStorage interface
    
    make them all accept a Dict[Sha1Git, xxx] as argument, ie:
    
    - remove support for Iterable[bytes] in revision_add, and
    - replace Iterable[bytes] by Dict[Sha1Git, bytes] for location_add
    
    Currently, the sha1 of location path in location_add() is not really
    used by any backend, so the computation of said hashed is a waste of
    resource, but it makes the API of this interface much more consistent
    which will be helpful for coming features (like kafka journal).

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/676/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/676/console

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 11 2022, 2:07 PM
Harbormaster failed remote builds in B32201: Diff 31249!
swh-public-ci added a comment.Oct 11 2022, 3:04 PM

Build is green

Patch application report for D8656 (id=31249)

Rebasing onto 6f4a193e90...

Current branch diff-target is up to date.
Changes applied before test
commit 6b539ecbe1673caf539e0636de76881a3f8ed171
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 7 14:51:09 2022 +0200

    Normalize _add() methods of the ProvenanceStorage interface
    
    make them all accept a Dict[Sha1Git, xxx] as argument, ie:
    
    - remove support for Iterable[bytes] in revision_add, and
    - replace Iterable[bytes] by Dict[Sha1Git, bytes] for location_add
    
    Currently, the sha1 of location path in location_add() is not really
    used by any backend, so the computation of said hashed is a waste of
    resource, but it makes the API of this interface much more consistent
    which will be helpful for coming features (like kafka journal).

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/679/ for more details.

Harbormaster completed remote builds in B32201: Diff 31249.Oct 11 2022, 3:04 PM
douardda requested review of this revision.Oct 11 2022, 3:04 PM
vlorentz accepted this revision.Oct 11 2022, 3:14 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/provenance/provenance.py
6

I'm not a huge fan of import ... from with stdlib modules in general, but here in particular there is the extra issue of sha1 being a common variable name

(and then you can undo the variable name change below)

swh/provenance/tests/test_provenance_storage.py
6

ditto

This revision is now accepted and ready to land.Oct 11 2022, 3:14 PM
douardda added inline comments.Oct 11 2022, 3:52 PM
swh/provenance/provenance.py
6

yeah I guess you are right, let me change this (thx)

douardda updated this revision to Diff 31270.Oct 11 2022, 4:05 PM

apply vlorentz' comments

douardda marked 2 inline comments as done.Oct 11 2022, 4:07 PM
swh-public-ci added a comment.Oct 11 2022, 4:11 PM

Build is green

Patch application report for D8656 (id=31270)

Rebasing onto 6f4a193e90...

Current branch diff-target is up to date.
Changes applied before test
commit 2bd74fc7d97d40d7132a6530cd0078e3ffb8c614
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 7 14:51:09 2022 +0200

    Normalize _add() methods of the ProvenanceStorage interface
    
    make them all accept a Dict[Sha1Git, xxx] as argument, ie:
    
    - remove support for Iterable[bytes] in revision_add, and
    - replace Iterable[bytes] by Dict[Sha1Git, bytes] for location_add
    
    Currently, the sha1 of location path in location_add() is not really
    used by any backend, so the computation of said hashed is a waste of
    resource, but it makes the API of this interface much more consistent
    which will be helpful for coming features (like kafka journal).

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/681/ for more details.

Harbormaster completed remote builds in B32221: Diff 31270.Oct 11 2022, 4:11 PM
douardda updated this revision to Diff 31273.Oct 11 2022, 4:38 PM

also rename ProvenanceInterface.directory_xxx_flattenned as directory_xxx_flattened

(included in this Diff by lazyness)

swh-public-ci added a comment.Oct 11 2022, 4:44 PM

Build is green

Patch application report for D8656 (id=31273)

Rebasing onto 6f4a193e90...

Current branch diff-target is up to date.
Changes applied before test
commit 7e6a62c990b76ac63ee53be1f4c1c147bba4b806
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Oct 11 16:30:46 2022 +0200

    Rename ProvenanceInterface.directory_xxx_flattenned as directory_xxx_flattened
    
    and fix all occurrences of the typo.

commit 2bd74fc7d97d40d7132a6530cd0078e3ffb8c614
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Oct 7 14:51:09 2022 +0200

    Normalize _add() methods of the ProvenanceStorage interface
    
    make them all accept a Dict[Sha1Git, xxx] as argument, ie:
    
    - remove support for Iterable[bytes] in revision_add, and
    - replace Iterable[bytes] by Dict[Sha1Git, bytes] for location_add
    
    Currently, the sha1 of location path in location_add() is not really
    used by any backend, so the computation of said hashed is a waste of
    resource, but it makes the API of this interface much more consistent
    which will be helpful for coming features (like kafka journal).

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/683/ for more details.

Harbormaster completed remote builds in B32224: Diff 31273.Oct 11 2022, 4:44 PM
Closed by commit rDPROV2bd74fc7d97d: Normalize _add() methods of the ProvenanceStorage interface (authored by douardda). · Explain WhyOct 12 2022, 4:07 PM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDPROV2bd74fc7d97d: Normalize _add() methods of the ProvenanceStorage interface.
douardda added a commit: rDPROV7e6a62c990b7: Rename ProvenanceInterface.directory_xxx_flattenned as directory_xxx_flattened.

Revision Contents
Changeset List

PathSize
swh/
provenance/
5 lines
storage/
10 lines
20 lines
rabbitmq/
1 line
2 lines
tests/
30 lines
6 lines

Diff 31301

View Options

swh/provenance/provenance.py

Loading...
View Options

swh/provenance/storage/interface.py

Loading...
View Options

swh/provenance/storage/postgresql.py

Loading...
View Options

swh/provenance/storage/rabbitmq/client.py

Loading...
View Options

swh/provenance/storage/rabbitmq/server.py

Loading...
View Options

swh/provenance/tests/test_provenance_storage.py

Loading...
View Options

swh/provenance/tests/test_revision_content_layer.py

Loading...
About · Developer information · Legal