Page MenuHomeSoftware Heritage
Differential D5985

Simplify history graph creation and origin-revision algorithm
ClosedPublic
Actions

Authored by aeviso on Jul 9 2021, 1:36 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Commits
rDPROV509280132cb3: Simplify history graph creation and origin-revision algorithm
rDPROV505869054bda: Add missing case to history graph yaml test file
Summary

Add missing case to history graph yaml test file

Fix typo too

Simplify history graph creation and origin-revision algorithm

Depends on D5973

Diff Detail

Event Timeline

aeviso created this revision.Jul 9 2021, 1:36 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 9 2021, 1:36 PM
aeviso mentioned this in D5986: Add provenance storage server/client classes.Jul 9 2021, 1:37 PM
swh-public-ci added a comment.Jul 9 2021, 1:41 PM

Build is green

Patch application report for D5985 (id=21580)

Could not rebase; Attempt merge onto 2e46f7c34b...

Updating 2e46f7c..9b67209
Fast-forward
 swh/provenance/__init__.py                         |  15 +-
 swh/provenance/archive.py                          |   5 +
 swh/provenance/backend.py                          | 324 ---------------
 swh/provenance/cli.py                              |  47 ++-
 swh/provenance/graph.py                            |  26 +-
 swh/provenance/interface.py                        | 318 +++++++++++++++
 swh/provenance/origin.py                           |  26 +-
 swh/provenance/postgresql/archive.py               |   5 +
 swh/provenance/postgresql/provenancedb_base.py     | 138 ++++---
 .../postgresql/provenancedb_with_path.py           |   7 +-
 .../postgresql/provenancedb_without_path.py        |   5 +
 swh/provenance/provenance.py                       | 450 ++++++++++++---------
 swh/provenance/revision.py                         |   7 +-
 swh/provenance/storage/archive.py                  |   5 +
 swh/provenance/tests/conftest.py                   |   2 +-
 swh/provenance/tests/data/generate_repo.py         |   2 +-
 .../data/history_graphs_with-merges_visits-01.yaml | 144 ++++++-
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |   2 +-
 swh/provenance/tests/test_isochrone_graph.py       |   2 +-
 swh/provenance/tests/test_provenance_db.py         |   2 +-
 swh/provenance/tests/test_provenance_heuristics.py | 193 ++++-----
 22 files changed, 985 insertions(+), 742 deletions(-)
 delete mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/interface.py
Changes applied before test
commit 9b67209cec29e0f8560028575bf410e7e0a3512d
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 12:07:07 2021 +0200

    Simplify history graph creation and origin-revision algorithm

commit 1091a7c966473ae42d7aac8514c47b9782082c55
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 10:59:12 2021 +0200

    Add missing case to history graph yaml test file
    
    Fix typo too

commit bcf19e3952ad841ed4bb4d78504ea70b1b4fb443
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:29:12 2021 +0200

    Improve configuration file format

commit 8563c9e2915fd785d831e27ca12ed1c896fd4c2e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:24:39 2021 +0200

    Add missing license string to several files

commit b37e8673c0d3111cc3a496733cb92c4affdad996
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 14:46:34 2021 +0200

    Reorganize code
    
    Move `ProvenanceInterface` and `ProvenanceStorageInterface` to `interface.py`.
    Rename `ProvenanceBackend` to `Provenance` and move it to `provenance.py`.

commit c246bd709270a6d5d407260f13cc666c18ab9819
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 13:48:50 2021 +0200

    Add `RevisionData` and `RelationData` to improve `ProvenanceStorageInterface`
    
    Get read of `Tuple`s in favour of more declarative objects for passing data.
    Update the usage of the involved methods as well.

commit 3003b15bf120734d8ee51472c594e117a6e9ea28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:30:34 2021 +0200

    Add parametrize `batch` option to heuristics test

commit 9c383756bc5f232a338060f2b6f8872980b71119
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:22:21 2021 +0200

    Add testing-specific methods to `ProvenanceStorageInterface`
    
    Refactor `provenance_heuristics` test so that it doesn't perform direct SQL queries,
    hence being able to test different backends in the future.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/253/ for more details.

Harbormaster completed remote builds in B22555: Diff 21580.Jul 9 2021, 1:41 PM
aeviso requested review of this revision.Jul 9 2021, 1:41 PM
aeviso updated this revision to Diff 21584.Jul 9 2021, 2:44 PM

rebase

aeviso added a child revision: D5986: Add provenance storage server/client classes.Jul 9 2021, 2:46 PM
swh-public-ci added a comment.Jul 9 2021, 2:53 PM

Build is green

Patch application report for D5985 (id=21584)

Could not rebase; Attempt merge onto 2e46f7c34b...

Updating 2e46f7c..f3bfea4
Fast-forward
 swh/provenance/__init__.py                         |  15 +-
 swh/provenance/archive.py                          |   5 +
 swh/provenance/backend.py                          | 324 ---------------
 swh/provenance/cli.py                              |  47 ++-
 swh/provenance/graph.py                            |  26 +-
 swh/provenance/interface.py                        | 303 ++++++++++++++
 swh/provenance/origin.py                           |  26 +-
 swh/provenance/postgresql/archive.py               |   5 +
 swh/provenance/postgresql/provenancedb_base.py     | 138 ++++---
 .../postgresql/provenancedb_with_path.py           |   7 +-
 .../postgresql/provenancedb_without_path.py        |   5 +
 swh/provenance/provenance.py                       | 450 ++++++++++++---------
 swh/provenance/revision.py                         |   7 +-
 swh/provenance/storage/archive.py                  |   5 +
 swh/provenance/tests/conftest.py                   |   2 +-
 swh/provenance/tests/data/generate_repo.py         |   2 +-
 .../data/history_graphs_with-merges_visits-01.yaml | 144 ++++++-
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |   2 +-
 swh/provenance/tests/test_isochrone_graph.py       |   2 +-
 swh/provenance/tests/test_provenance_db.py         |   2 +-
 swh/provenance/tests/test_provenance_heuristics.py | 193 ++++-----
 22 files changed, 970 insertions(+), 742 deletions(-)
 delete mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/interface.py
Changes applied before test
commit f3bfea485dc75584755fc6026c9691be73276fc2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 12:07:07 2021 +0200

    Simplify history graph creation and origin-revision algorithm

commit 6b0eea9e55970925d2eec33fdb1b047f6db4a449
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 10:59:12 2021 +0200

    Add missing case to history graph yaml test file
    
    Fix typo too

commit cbd38f4a5b51ede687b4ad713df3d26d2bded0b2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:29:12 2021 +0200

    Improve configuration file format

commit 8c240c7ff2f8003fc01c1ffe18d5badf282f2de0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:24:39 2021 +0200

    Add missing license string to several files

commit 1282b24695016702a59b77075ecc9c27e6f5063f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 14:46:34 2021 +0200

    Reorganize code
    
    Move `ProvenanceInterface` and `ProvenanceStorageInterface` to `interface.py`.
    Rename `ProvenanceBackend` to `Provenance` and move it to `provenance.py`.

commit c96beb57884d19c7fd1fa74aa366f472187419ff
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 13:48:50 2021 +0200

    Add `RevisionData` and `RelationData` to improve `ProvenanceStorageInterface`
    
    Get read of `Tuple`s in favour of more declarative objects for passing data, and
    update the usage of the involved methods.
    
    Turn `ProvenanceResult` into a dataclass as well.

commit 3003b15bf120734d8ee51472c594e117a6e9ea28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:30:34 2021 +0200

    Add parametrize `batch` option to heuristics test

commit 9c383756bc5f232a338060f2b6f8872980b71119
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:22:21 2021 +0200

    Add testing-specific methods to `ProvenanceStorageInterface`
    
    Refactor `provenance_heuristics` test so that it doesn't perform direct SQL queries,
    hence being able to test different backends in the future.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/257/ for more details.

Harbormaster completed remote builds in B22559: Diff 21584.Jul 9 2021, 2:53 PM
aeviso removed a child revision: D5986: Add provenance storage server/client classes.Jul 12 2021, 11:06 AM
aeviso updated this revision to Diff 21593.Jul 12 2021, 11:08 AM
aeviso added a child revision: D5986: Add provenance storage server/client classes.

rebase

aeviso removed a parent revision: D5973: Reorganize code.Jul 12 2021, 11:10 AM
aeviso removed a child revision: D5986: Add provenance storage server/client classes.
aeviso added a parent revision: D5986: Add provenance storage server/client classes.
swh-public-ci added a comment.Jul 12 2021, 11:22 AM

Build is green

Patch application report for D5985 (id=21593)

Could not rebase; Attempt merge onto 2e46f7c34b...

Updating 2e46f7c..5092801
Fast-forward
 swh/provenance/__init__.py                         |  30 +-
 swh/provenance/api/__init__.py                     |   0
 swh/provenance/api/client.py                       |  17 +
 swh/provenance/api/serializers.py                  |  48 +++
 swh/provenance/api/server.py                       | 143 +++++++
 swh/provenance/archive.py                          |   5 +
 swh/provenance/backend.py                          | 324 ---------------
 swh/provenance/cli.py                              |  47 ++-
 swh/provenance/graph.py                            |  26 +-
 swh/provenance/interface.py                        | 322 +++++++++++++++
 swh/provenance/origin.py                           |  26 +-
 swh/provenance/postgresql/archive.py               |   5 +
 swh/provenance/postgresql/provenancedb_base.py     | 146 ++++---
 .../postgresql/provenancedb_with_path.py           |   7 +-
 .../postgresql/provenancedb_without_path.py        |   5 +
 swh/provenance/provenance.py                       | 450 ++++++++++++---------
 swh/provenance/revision.py                         |   7 +-
 swh/provenance/storage/archive.py                  |   5 +
 swh/provenance/tests/conftest.py                   |  60 ++-
 swh/provenance/tests/data/generate_repo.py         |   2 +-
 .../data/history_graphs_with-merges_visits-01.yaml | 144 ++++++-
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |   2 +-
 swh/provenance/tests/test_isochrone_graph.py       |   2 +-
 swh/provenance/tests/test_provenance_db.py         |  18 +-
 swh/provenance/tests/test_provenance_heuristics.py | 207 ++++------
 swh/provenance/tests/test_provenance_storage.py    |  46 +++
 27 files changed, 1312 insertions(+), 784 deletions(-)
 create mode 100644 swh/provenance/api/__init__.py
 create mode 100644 swh/provenance/api/client.py
 create mode 100644 swh/provenance/api/serializers.py
 create mode 100644 swh/provenance/api/server.py
 delete mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/interface.py
 create mode 100644 swh/provenance/tests/test_provenance_storage.py
Changes applied before test
commit 509280132cb35df1432905097dafb24870690ee1
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 12:07:07 2021 +0200

    Simplify history graph creation and origin-revision algorithm

commit 505869054bda044c72d1a907631db552d173870f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 10:59:12 2021 +0200

    Add missing case to history graph yaml test file
    
    Fix typo too

commit 1ae32c0a61e1ac12e934544cd783c1319b891827
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 13:06:35 2021 +0200

    Add test to check that all storage implementations ara compliant with `ProvenanceStorageInterface`

commit e9d4009d0c78d79d299e6a981178e3131da67d9d
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 13:05:54 2021 +0200

    Add provenance storage server/client classes

commit 00fc11dfb70d5a3b2d8218350c46144f096524c5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 9 12:57:06 2021 +0200

    Remove `raise_on_commit` from `ProvenanceStorageInterface`
    
    It is now an attributte of `ProvenanceDBBase`

commit 06d0487cc7f4bdb0f0ca2cb9934659f9307a1abb
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 12:09:29 2021 +0200

    Add `with_path` method to the `ProvenanceStorageInterface`

commit cbd38f4a5b51ede687b4ad713df3d26d2bded0b2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:29:12 2021 +0200

    Improve configuration file format

commit 8c240c7ff2f8003fc01c1ffe18d5badf282f2de0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 8 17:24:39 2021 +0200

    Add missing license string to several files

commit 1282b24695016702a59b77075ecc9c27e6f5063f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 14:46:34 2021 +0200

    Reorganize code
    
    Move `ProvenanceInterface` and `ProvenanceStorageInterface` to `interface.py`.
    Rename `ProvenanceBackend` to `Provenance` and move it to `provenance.py`.

commit c96beb57884d19c7fd1fa74aa366f472187419ff
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 13:48:50 2021 +0200

    Add `RevisionData` and `RelationData` to improve `ProvenanceStorageInterface`
    
    Get read of `Tuple`s in favour of more declarative objects for passing data, and
    update the usage of the involved methods.
    
    Turn `ProvenanceResult` into a dataclass as well.

commit 3003b15bf120734d8ee51472c594e117a6e9ea28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:30:34 2021 +0200

    Add parametrize `batch` option to heuristics test

commit 9c383756bc5f232a338060f2b6f8872980b71119
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 6 12:22:21 2021 +0200

    Add testing-specific methods to `ProvenanceStorageInterface`
    
    Refactor `provenance_heuristics` test so that it doesn't perform direct SQL queries,
    hence being able to test different backends in the future.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/262/ for more details.

Harbormaster completed remote builds in B22572: Diff 21593.Jul 12 2021, 11:22 AM
aeviso added a child revision: D6026: Add test for origin-revision layer.Jul 27 2021, 12:40 PM
douardda accepted this revision.Jul 27 2021, 6:15 PM
This revision is now accepted and ready to land.Jul 27 2021, 6:15 PM
Closed by commit rDPROV505869054bda: Add missing case to history graph yaml test file (authored by aeviso). · Explain WhyJul 28 2021, 11:30 AM
This revision was automatically updated to reflect the committed changes.
aeviso added a commit: rDPROV505869054bda: Add missing case to history graph yaml test file.
aeviso added a commit: rDPROV509280132cb3: Simplify history graph creation and origin-revision algorithm.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
provenance/
19 lines100%
Loading...
19 lines97%
Loading...
tests/
data/
2 lines21%
Loading...
144 lines--

Diff 21580

View Options

swh/provenance/graph.py

Loading...
View Options

swh/provenance/origin.py

Loading...
View Options

swh/provenance/tests/data/generate_repo.py

Loading...
View Options

swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml

Loading...
About · Developer information · Legal