Page MenuHomeSoftware Heritage
Differential D5986

Add provenance storage server/client classes
ClosedPublic
Actions

Authored by aeviso on Jul 9 2021, 1:37 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDPROV06d0487cc7f4: Add `with_path` method to the `ProvenanceStorageInterface`
rDPROV00fc11dfb70d: Remove `raise_on_commit` from `ProvenanceStorageInterface`
rDPROVe9d4009d0c78: Add provenance storage server/client classes
rDPROV1ae32c0a61e1: Add test to check that all storage implementations ara compliant with…
Summary

Add with_path method to the ProvenanceStorageInterface and turn custom result classes into dataclasses

Remove raise_on_commit from ProvenanceStorageInterface

It is now an attributte of ProvenanceDBBase

Add provenance storage server/client classes

Add test to check that all storage implementations ara compliant with ProvenanceStorageInterface

Depends on D5985

Diff Detail

Event Timeline

aeviso created this revision.Jul 9 2021, 1:37 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 9 2021, 1:37 PM
swh-public-ci added a comment.Jul 9 2021, 1:42 PM

Build has FAILED

Patch application report for D5986 (id=21581)

Could not rebase; Attempt merge onto 2e46f7c34b...

Updating 2e46f7c..5739e4b
Fast-forward
 swh/provenance/__init__.py                         |  28 +-
 swh/provenance/api/__init__.py                     |   0
 swh/provenance/api/client.py                       |  17 +
 swh/provenance/api/serializers.py                  |  48 +++
 swh/provenance/api/server.py                       | 148 +++++++
 swh/provenance/archive.py                          |   5 +
 swh/provenance/backend.py                          | 324 ---------------
 swh/provenance/cli.py                              |  47 ++-
 swh/provenance/graph.py                            |  26 +-
 swh/provenance/interface.py                        | 322 +++++++++++++++
 swh/provenance/origin.py                           |  26 +-
 swh/provenance/postgresql/archive.py               |   5 +
 swh/provenance/postgresql/provenancedb_base.py     | 146 ++++---
 .../postgresql/provenancedb_with_path.py           |   7 +-
 .../postgresql/provenancedb_without_path.py        |   5 +
 swh/provenance/provenance.py                       | 450 ++++++++++++---------
 swh/provenance/revision.py                         |   7 +-
 swh/provenance/storage/archive.py                  |   5 +
 swh/provenance/tests/conftest.py                   |  59 ++-
 swh/provenance/tests/data/generate_repo.py         |   2 +-
 .../data/history_graphs_with-merges_visits-01.yaml | 144 ++++++-
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |   2 +-
 swh/provenance/tests/test_isochrone_graph.py       |   2 +-
 swh/provenance/tests/test_provenance_db.py         |  18 +-
 swh/provenance/tests/test_provenance_heuristics.py | 207 ++++------
 swh/provenance/tests/test_provenance_storage.py    |  46 +++
 27 files changed, 1314 insertions(+), 784 deletions(-)
 create mode 100644 swh/provenance/api/__init__.py
 create mode 100644 swh/provenance/api/client.py
 create mode 100644 swh/provenance/api/serializers.py
 create mode 100644 swh/provenance/api/server.py
 delete mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/interface.py
 create mode 100644 swh/provenance/tests/test_provenance_storage.py
Changes applied before test
commit 5739e4bd2c5837d0d455100ec6014a018df5660f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 13:06:35 2021 +0200

    Add test to check that all storage implementations ara compliant with `ProvenanceStorageInterface`

commit 7941be0ed8934ebc5268270c70cdffc2ef53c04b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 13:05:54 2021 +0200

    Add provenance storage server/client classes

commit 02b39f704f4baa7d29f2728d24c36ecd76562d8e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 12:57:06 2021 +0200

    Remove `raise_on_commit` from `ProvenanceStorageInterface`
    
    It is now an attributte of `ProvenanceDBBase`

commit 036e00fe8eac64e86c13df1eb1dabd82df5f68a4
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 12:09:29 2021 +0200

    Add `with_path` method to the `ProvenanceStorageInterface` and turn custom result classes into `dataclasses`

commit 9b67209cec29e0f8560028575bf410e7e0a3512d
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 12:07:07 2021 +0200

    Simplify history graph creation and origin-revision algorithm

commit 1091a7c966473ae42d7aac8514c47b9782082c55
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 10:59:12 2021 +0200

    Add missing case to history graph yaml test file
    
    Fix typo too

commit bcf19e3952ad841ed4bb4d78504ea70b1b4fb443
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:29:12 2021 +0200

    Improve configuration file format

commit 8563c9e2915fd785d831e27ca12ed1c896fd4c2e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:24:39 2021 +0200

    Add missing license string to several files

commit b37e8673c0d3111cc3a496733cb92c4affdad996
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 14:46:34 2021 +0200

    Reorganize code
    
    Move `ProvenanceInterface` and `ProvenanceStorageInterface` to `interface.py`.
    Rename `ProvenanceBackend` to `Provenance` and move it to `provenance.py`.

commit c246bd709270a6d5d407260f13cc666c18ab9819
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 13:48:50 2021 +0200

    Add `RevisionData` and `RelationData` to improve `ProvenanceStorageInterface`
    
    Get read of `Tuple`s in favour of more declarative objects for passing data.
    Update the usage of the involved methods as well.

commit 3003b15bf120734d8ee51472c594e117a6e9ea28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:30:34 2021 +0200

    Add parametrize `batch` option to heuristics test

commit 9c383756bc5f232a338060f2b6f8872980b71119
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:22:21 2021 +0200

    Add testing-specific methods to `ProvenanceStorageInterface`
    
    Refactor `provenance_heuristics` test so that it doesn't perform direct SQL queries,
    hence being able to test different backends in the future.

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/254/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/254/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jul 9 2021, 1:42 PM
Harbormaster failed remote builds in B22556: Diff 21581!
aeviso updated this revision to Diff 21585.Jul 9 2021, 2:46 PM

rebase

aeviso edited the summary of this revision. (Show Details)Jul 9 2021, 2:46 PM
aeviso added a parent revision: D5985: Simplify history graph creation and origin-revision algorithm.
aeviso updated this revision to Diff 21586.Jul 9 2021, 2:56 PM

rebase

swh-public-ci added a comment.Jul 9 2021, 3:00 PM

Build is green

Patch application report for D5986 (id=21585)

Could not rebase; Attempt merge onto 2e46f7c34b...

Updating 2e46f7c..7ed8710
Fast-forward
 swh/provenance/__init__.py                         |  30 +-
 swh/provenance/api/__init__.py                     |   0
 swh/provenance/api/client.py                       |  17 +
 swh/provenance/api/serializers.py                  |  48 +++
 swh/provenance/api/server.py                       | 148 +++++++
 swh/provenance/archive.py                          |   5 +
 swh/provenance/backend.py                          | 324 ---------------
 swh/provenance/cli.py                              |  47 ++-
 swh/provenance/graph.py                            |  26 +-
 swh/provenance/interface.py                        | 322 +++++++++++++++
 swh/provenance/origin.py                           |  26 +-
 swh/provenance/postgresql/archive.py               |   5 +
 swh/provenance/postgresql/provenancedb_base.py     | 146 ++++---
 .../postgresql/provenancedb_with_path.py           |   7 +-
 .../postgresql/provenancedb_without_path.py        |   5 +
 swh/provenance/provenance.py                       | 450 ++++++++++++---------
 swh/provenance/revision.py                         |   7 +-
 swh/provenance/storage/archive.py                  |   5 +
 swh/provenance/tests/conftest.py                   |  60 ++-
 swh/provenance/tests/data/generate_repo.py         |   2 +-
 .../data/history_graphs_with-merges_visits-01.yaml | 144 ++++++-
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |   2 +-
 swh/provenance/tests/test_isochrone_graph.py       |   2 +-
 swh/provenance/tests/test_provenance_db.py         |  18 +-
 swh/provenance/tests/test_provenance_heuristics.py | 207 ++++------
 swh/provenance/tests/test_provenance_storage.py    |  46 +++
 27 files changed, 1317 insertions(+), 784 deletions(-)
 create mode 100644 swh/provenance/api/__init__.py
 create mode 100644 swh/provenance/api/client.py
 create mode 100644 swh/provenance/api/serializers.py
 create mode 100644 swh/provenance/api/server.py
 delete mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/interface.py
 create mode 100644 swh/provenance/tests/test_provenance_storage.py
Changes applied before test
commit 7ed8710c4bc280e45884cb3fbd85d3b60438d271
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 13:06:35 2021 +0200

    Add test to check that all storage implementations ara compliant with `ProvenanceStorageInterface`

commit fcf5fc7f4c4059bfb47d7cc18eaee0ca79d1b179
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 13:05:54 2021 +0200

    Add provenance storage server/client classes

commit bb23b306c2c756bbd61c23e5ee983271d370d7a5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 12:57:06 2021 +0200

    Remove `raise_on_commit` from `ProvenanceStorageInterface`
    
    It is now an attributte of `ProvenanceDBBase`

commit 613a4e43ca994fbae790abacfe4ab871ee537319
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 12:09:29 2021 +0200

    Add `with_path` method to the `ProvenanceStorageInterface`

commit f3bfea485dc75584755fc6026c9691be73276fc2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 12:07:07 2021 +0200

    Simplify history graph creation and origin-revision algorithm

commit 6b0eea9e55970925d2eec33fdb1b047f6db4a449
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 10:59:12 2021 +0200

    Add missing case to history graph yaml test file
    
    Fix typo too

commit cbd38f4a5b51ede687b4ad713df3d26d2bded0b2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:29:12 2021 +0200

    Improve configuration file format

commit 8c240c7ff2f8003fc01c1ffe18d5badf282f2de0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:24:39 2021 +0200

    Add missing license string to several files

commit 1282b24695016702a59b77075ecc9c27e6f5063f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 14:46:34 2021 +0200

    Reorganize code
    
    Move `ProvenanceInterface` and `ProvenanceStorageInterface` to `interface.py`.
    Rename `ProvenanceBackend` to `Provenance` and move it to `provenance.py`.

commit c96beb57884d19c7fd1fa74aa366f472187419ff
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 13:48:50 2021 +0200

    Add `RevisionData` and `RelationData` to improve `ProvenanceStorageInterface`
    
    Get read of `Tuple`s in favour of more declarative objects for passing data, and
    update the usage of the involved methods.
    
    Turn `ProvenanceResult` into a dataclass as well.

commit 3003b15bf120734d8ee51472c594e117a6e9ea28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:30:34 2021 +0200

    Add parametrize `batch` option to heuristics test

commit 9c383756bc5f232a338060f2b6f8872980b71119
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:22:21 2021 +0200

    Add testing-specific methods to `ProvenanceStorageInterface`
    
    Refactor `provenance_heuristics` test so that it doesn't perform direct SQL queries,
    hence being able to test different backends in the future.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/258/ for more details.

Harbormaster completed remote builds in B22560: Diff 21585.Jul 9 2021, 3:00 PM
swh-public-ci added a comment.Jul 9 2021, 3:07 PM

Build is green

Patch application report for D5986 (id=21586)

Could not rebase; Attempt merge onto 2e46f7c34b...

Updating 2e46f7c..990dc12
Fast-forward
 swh/provenance/__init__.py                         |  30 +-
 swh/provenance/api/__init__.py                     |   0
 swh/provenance/api/client.py                       |  17 +
 swh/provenance/api/serializers.py                  |  48 +++
 swh/provenance/api/server.py                       | 148 +++++++
 swh/provenance/archive.py                          |   5 +
 swh/provenance/backend.py                          | 324 ---------------
 swh/provenance/cli.py                              |  47 ++-
 swh/provenance/graph.py                            |  26 +-
 swh/provenance/interface.py                        | 322 +++++++++++++++
 swh/provenance/origin.py                           |  26 +-
 swh/provenance/postgresql/archive.py               |   5 +
 swh/provenance/postgresql/provenancedb_base.py     | 146 ++++---
 .../postgresql/provenancedb_with_path.py           |   7 +-
 .../postgresql/provenancedb_without_path.py        |   5 +
 swh/provenance/provenance.py                       | 450 ++++++++++++---------
 swh/provenance/revision.py                         |   7 +-
 swh/provenance/storage/archive.py                  |   5 +
 swh/provenance/tests/conftest.py                   |  60 ++-
 swh/provenance/tests/data/generate_repo.py         |   2 +-
 .../data/history_graphs_with-merges_visits-01.yaml | 144 ++++++-
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |   2 +-
 swh/provenance/tests/test_isochrone_graph.py       |   2 +-
 swh/provenance/tests/test_provenance_db.py         |  18 +-
 swh/provenance/tests/test_provenance_heuristics.py | 207 ++++------
 swh/provenance/tests/test_provenance_storage.py    |  46 +++
 27 files changed, 1317 insertions(+), 784 deletions(-)
 create mode 100644 swh/provenance/api/__init__.py
 create mode 100644 swh/provenance/api/client.py
 create mode 100644 swh/provenance/api/serializers.py
 create mode 100644 swh/provenance/api/server.py
 delete mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/interface.py
 create mode 100644 swh/provenance/tests/test_provenance_storage.py
Changes applied before test
commit 990dc123df31af7418eb06afc63fe9f44450f823
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 13:06:35 2021 +0200

    Add test to check that all storage implementations ara compliant with `ProvenanceStorageInterface`

commit 5856d79f8c970faf44ee8f41e23792bafb620854
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 13:05:54 2021 +0200

    Add provenance storage server/client classes

commit 9196c88e918745f48ab514de554e63b02912d15f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 12:57:06 2021 +0200

    Remove `raise_on_commit` from `ProvenanceStorageInterface`
    
    It is now an attributte of `ProvenanceDBBase`

commit 613a4e43ca994fbae790abacfe4ab871ee537319
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 12:09:29 2021 +0200

    Add `with_path` method to the `ProvenanceStorageInterface`

commit f3bfea485dc75584755fc6026c9691be73276fc2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 12:07:07 2021 +0200

    Simplify history graph creation and origin-revision algorithm

commit 6b0eea9e55970925d2eec33fdb1b047f6db4a449
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 10:59:12 2021 +0200

    Add missing case to history graph yaml test file
    
    Fix typo too

commit cbd38f4a5b51ede687b4ad713df3d26d2bded0b2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:29:12 2021 +0200

    Improve configuration file format

commit 8c240c7ff2f8003fc01c1ffe18d5badf282f2de0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:24:39 2021 +0200

    Add missing license string to several files

commit 1282b24695016702a59b77075ecc9c27e6f5063f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 14:46:34 2021 +0200

    Reorganize code
    
    Move `ProvenanceInterface` and `ProvenanceStorageInterface` to `interface.py`.
    Rename `ProvenanceBackend` to `Provenance` and move it to `provenance.py`.

commit c96beb57884d19c7fd1fa74aa366f472187419ff
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 13:48:50 2021 +0200

    Add `RevisionData` and `RelationData` to improve `ProvenanceStorageInterface`
    
    Get read of `Tuple`s in favour of more declarative objects for passing data, and
    update the usage of the involved methods.
    
    Turn `ProvenanceResult` into a dataclass as well.

commit 3003b15bf120734d8ee51472c594e117a6e9ea28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:30:34 2021 +0200

    Add parametrize `batch` option to heuristics test

commit 9c383756bc5f232a338060f2b6f8872980b71119
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:22:21 2021 +0200

    Add testing-specific methods to `ProvenanceStorageInterface`
    
    Refactor `provenance_heuristics` test so that it doesn't perform direct SQL queries,
    hence being able to test different backends in the future.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/259/ for more details.

Harbormaster completed remote builds in B22561: Diff 21586.Jul 9 2021, 3:07 PM
aeviso requested review of this revision.Jul 9 2021, 3:07 PM
vlorentz accepted this revision.Jul 9 2021, 5:34 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/provenance/api/server.py
143–145

remove this?

This revision is now accepted and ready to land.Jul 9 2021, 5:34 PM
aeviso marked an inline comment as done.Jul 10 2021, 7:38 PM
aeviso added inline comments.
swh/provenance/api/server.py
143–145

Yeah, clearly. I've totally missed that. Thanks!

aeviso updated this revision to Diff 21591.Jul 10 2021, 7:52 PM
aeviso marked an inline comment as done.

rebase

swh-public-ci added a comment.Jul 10 2021, 7:59 PM

Build is green

Patch application report for D5986 (id=21591)

Could not rebase; Attempt merge onto 2e46f7c34b...

Updating 2e46f7c..01f68ac
Fast-forward
 swh/provenance/__init__.py                         |  30 +-
 swh/provenance/api/__init__.py                     |   0
 swh/provenance/api/client.py                       |  17 +
 swh/provenance/api/serializers.py                  |  48 +++
 swh/provenance/api/server.py                       | 143 +++++++
 swh/provenance/archive.py                          |   5 +
 swh/provenance/backend.py                          | 324 ---------------
 swh/provenance/cli.py                              |  47 ++-
 swh/provenance/graph.py                            |  26 +-
 swh/provenance/interface.py                        | 322 +++++++++++++++
 swh/provenance/origin.py                           |  26 +-
 swh/provenance/postgresql/archive.py               |   5 +
 swh/provenance/postgresql/provenancedb_base.py     | 146 ++++---
 .../postgresql/provenancedb_with_path.py           |   7 +-
 .../postgresql/provenancedb_without_path.py        |   5 +
 swh/provenance/provenance.py                       | 450 ++++++++++++---------
 swh/provenance/revision.py                         |   7 +-
 swh/provenance/storage/archive.py                  |   5 +
 swh/provenance/tests/conftest.py                   |  60 ++-
 swh/provenance/tests/data/generate_repo.py         |   2 +-
 .../data/history_graphs_with-merges_visits-01.yaml | 144 ++++++-
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |   2 +-
 swh/provenance/tests/test_isochrone_graph.py       |   2 +-
 swh/provenance/tests/test_provenance_db.py         |  18 +-
 swh/provenance/tests/test_provenance_heuristics.py | 207 ++++------
 swh/provenance/tests/test_provenance_storage.py    |  46 +++
 27 files changed, 1312 insertions(+), 784 deletions(-)
 create mode 100644 swh/provenance/api/__init__.py
 create mode 100644 swh/provenance/api/client.py
 create mode 100644 swh/provenance/api/serializers.py
 create mode 100644 swh/provenance/api/server.py
 delete mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/interface.py
 create mode 100644 swh/provenance/tests/test_provenance_storage.py
Changes applied before test
commit 01f68ac59b55aede572c9de28b500a472e29d46b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 13:06:35 2021 +0200

    Add test to check that all storage implementations ara compliant with `ProvenanceStorageInterface`

commit b30e6dc21b0bdb5d53d7f3f7d698bc4c9662e890
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 13:05:54 2021 +0200

    Add provenance storage server/client classes

commit 9196c88e918745f48ab514de554e63b02912d15f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 12:57:06 2021 +0200

    Remove `raise_on_commit` from `ProvenanceStorageInterface`
    
    It is now an attributte of `ProvenanceDBBase`

commit 613a4e43ca994fbae790abacfe4ab871ee537319
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 12:09:29 2021 +0200

    Add `with_path` method to the `ProvenanceStorageInterface`

commit f3bfea485dc75584755fc6026c9691be73276fc2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 12:07:07 2021 +0200

    Simplify history graph creation and origin-revision algorithm

commit 6b0eea9e55970925d2eec33fdb1b047f6db4a449
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 10:59:12 2021 +0200

    Add missing case to history graph yaml test file
    
    Fix typo too

commit cbd38f4a5b51ede687b4ad713df3d26d2bded0b2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:29:12 2021 +0200

    Improve configuration file format

commit 8c240c7ff2f8003fc01c1ffe18d5badf282f2de0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:24:39 2021 +0200

    Add missing license string to several files

commit 1282b24695016702a59b77075ecc9c27e6f5063f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 14:46:34 2021 +0200

    Reorganize code
    
    Move `ProvenanceInterface` and `ProvenanceStorageInterface` to `interface.py`.
    Rename `ProvenanceBackend` to `Provenance` and move it to `provenance.py`.

commit c96beb57884d19c7fd1fa74aa366f472187419ff
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 13:48:50 2021 +0200

    Add `RevisionData` and `RelationData` to improve `ProvenanceStorageInterface`
    
    Get read of `Tuple`s in favour of more declarative objects for passing data, and
    update the usage of the involved methods.
    
    Turn `ProvenanceResult` into a dataclass as well.

commit 3003b15bf120734d8ee51472c594e117a6e9ea28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:30:34 2021 +0200

    Add parametrize `batch` option to heuristics test

commit 9c383756bc5f232a338060f2b6f8872980b71119
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:22:21 2021 +0200

    Add testing-specific methods to `ProvenanceStorageInterface`
    
    Refactor `provenance_heuristics` test so that it doesn't perform direct SQL queries,
    hence being able to test different backends in the future.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/260/ for more details.

Harbormaster completed remote builds in B22570: Diff 21591.Jul 10 2021, 7:59 PM
aeviso edited parent revisions, added: D5973: Reorganize code; removed: D5985: Simplify history graph creation and origin-revision algorithm.Jul 12 2021, 11:06 AM
aeviso updated this revision to Diff 21592.Jul 12 2021, 11:08 AM
aeviso added a parent revision: D5985: Simplify history graph creation and origin-revision algorithm.

rebase

aeviso removed a parent revision: D5985: Simplify history graph creation and origin-revision algorithm.Jul 12 2021, 11:10 AM
aeviso added a child revision: D5985: Simplify history graph creation and origin-revision algorithm.
swh-public-ci added a comment.Jul 12 2021, 11:15 AM

Build is green

Patch application report for D5986 (id=21592)

Could not rebase; Attempt merge onto 2e46f7c34b...

Updating 2e46f7c..1ae32c0
Fast-forward
 swh/provenance/__init__.py                         |  30 +-
 swh/provenance/api/__init__.py                     |   0
 swh/provenance/api/client.py                       |  17 +
 swh/provenance/api/serializers.py                  |  48 +++
 swh/provenance/api/server.py                       | 143 +++++++
 swh/provenance/archive.py                          |   5 +
 swh/provenance/backend.py                          | 324 ---------------
 swh/provenance/cli.py                              |  47 ++-
 swh/provenance/graph.py                            |   7 +-
 swh/provenance/interface.py                        | 322 +++++++++++++++
 swh/provenance/origin.py                           |   7 +-
 swh/provenance/postgresql/archive.py               |   5 +
 swh/provenance/postgresql/provenancedb_base.py     | 146 ++++---
 .../postgresql/provenancedb_with_path.py           |   7 +-
 .../postgresql/provenancedb_without_path.py        |   5 +
 swh/provenance/provenance.py                       | 450 ++++++++++++---------
 swh/provenance/revision.py                         |   7 +-
 swh/provenance/storage/archive.py                  |   5 +
 swh/provenance/tests/conftest.py                   |  60 ++-
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |   2 +-
 swh/provenance/tests/test_isochrone_graph.py       |   2 +-
 swh/provenance/tests/test_provenance_db.py         |  18 +-
 swh/provenance/tests/test_provenance_heuristics.py | 207 ++++------
 swh/provenance/tests/test_provenance_storage.py    |  46 +++
 25 files changed, 1154 insertions(+), 758 deletions(-)
 create mode 100644 swh/provenance/api/__init__.py
 create mode 100644 swh/provenance/api/client.py
 create mode 100644 swh/provenance/api/serializers.py
 create mode 100644 swh/provenance/api/server.py
 delete mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/interface.py
 create mode 100644 swh/provenance/tests/test_provenance_storage.py
Changes applied before test
commit 1ae32c0a61e1ac12e934544cd783c1319b891827
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 13:06:35 2021 +0200

    Add test to check that all storage implementations ara compliant with `ProvenanceStorageInterface`

commit e9d4009d0c78d79d299e6a981178e3131da67d9d
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 13:05:54 2021 +0200

    Add provenance storage server/client classes

commit 00fc11dfb70d5a3b2d8218350c46144f096524c5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 12:57:06 2021 +0200

    Remove `raise_on_commit` from `ProvenanceStorageInterface`
    
    It is now an attributte of `ProvenanceDBBase`

commit 06d0487cc7f4bdb0f0ca2cb9934659f9307a1abb
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 12:09:29 2021 +0200

    Add `with_path` method to the `ProvenanceStorageInterface`

commit cbd38f4a5b51ede687b4ad713df3d26d2bded0b2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:29:12 2021 +0200

    Improve configuration file format

commit 8c240c7ff2f8003fc01c1ffe18d5badf282f2de0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:24:39 2021 +0200

    Add missing license string to several files

commit 1282b24695016702a59b77075ecc9c27e6f5063f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 14:46:34 2021 +0200

    Reorganize code
    
    Move `ProvenanceInterface` and `ProvenanceStorageInterface` to `interface.py`.
    Rename `ProvenanceBackend` to `Provenance` and move it to `provenance.py`.

commit c96beb57884d19c7fd1fa74aa366f472187419ff
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 13:48:50 2021 +0200

    Add `RevisionData` and `RelationData` to improve `ProvenanceStorageInterface`
    
    Get read of `Tuple`s in favour of more declarative objects for passing data, and
    update the usage of the involved methods.
    
    Turn `ProvenanceResult` into a dataclass as well.

commit 3003b15bf120734d8ee51472c594e117a6e9ea28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:30:34 2021 +0200

    Add parametrize `batch` option to heuristics test

commit 9c383756bc5f232a338060f2b6f8872980b71119
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:22:21 2021 +0200

    Add testing-specific methods to `ProvenanceStorageInterface`
    
    Refactor `provenance_heuristics` test so that it doesn't perform direct SQL queries,
    hence being able to test different backends in the future.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/261/ for more details.

Harbormaster completed remote builds in B22571: Diff 21592.Jul 12 2021, 11:15 AM
Closed by commit rDPROV1ae32c0a61e1: Add test to check that all storage implementations ara compliant with… (authored by aeviso). · Explain WhyJul 12 2021, 11:19 AM
This revision was automatically updated to reflect the committed changes.
aeviso added a commit: rDPROV1ae32c0a61e1: Add test to check that all storage implementations ara compliant with….
aeviso added a commit: rDPROVe9d4009d0c78: Add provenance storage server/client classes.
aeviso added a commit: rDPROV00fc11dfb70d: Remove `raise_on_commit` from `ProvenanceStorageInterface`.
aeviso added a commit: rDPROV06d0487cc7f4: Add `with_path` method to the `ProvenanceStorageInterface`.

Revision Contents
Changeset List

PathSize
swh/
provenance/
13 lines
api/
17 lines
48 lines
148 lines
66 lines
postgresql/
8 lines
tests/
57 lines
16 lines
16 lines
46 lines

Diff 21581

View Options

swh/provenance/__init__.py

Loading...
View Options

swh/provenance/api/__init__.py

Loading...
View Options

swh/provenance/api/client.py

Loading...
View Options

swh/provenance/api/serializers.py

Loading...
View Options

swh/provenance/api/server.py

Loading...
View Options

swh/provenance/interface.py

Loading...
View Options

swh/provenance/postgresql/provenancedb_base.py

Loading...
View Options

swh/provenance/tests/conftest.py

Loading...
View Options

swh/provenance/tests/test_provenance_db.py

Loading...
View Options

swh/provenance/tests/test_provenance_heuristics.py

Loading...
View Options

swh/provenance/tests/test_provenance_storage.py

Loading...
About · Developer information · Legal