Page MenuHomeSoftware Heritage
Differential D6026

Add test for origin-revision layer
ClosedPublic
Actions

Authored by aeviso on Jul 27 2021, 12:40 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Commits
rDPROV91e4dcd4148d: Add some missing type annotations and remove already done TODO's
rDPROVbe098b32d8fb: Add test for origin-revision layer
rDPROVc58d170f7766: Fix type conversion when retrieving origins' urls from PostgreSQL backend
Summary

Rename revision-content layer test as well, for consistency.

Depends on D5985

Diff Detail

Unit TestsFailed
View All

TimeTest
716 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.provenance.tests.test_origin_revision_layer::test_origin_revision_layer[local-archive-with-path-with-merges-visits-01]
provenance = <swh.provenance.provenance.Provenance object at 0x7f1c15177470> swh_storage = <swh.storage.postgresql.storage.Storage object at 0x7f1c15045940> archive = <swh.provenance.storage.archive.ArchiveStorage object at 0x7f1c1504af98>
601 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.provenance.tests.test_origin_revision_layer::test_origin_revision_layer[local-archive-without-path-with-merges-visits-01]
provenance = <swh.provenance.provenance.Provenance object at 0x7f1c15128470> swh_storage = <swh.storage.postgresql.storage.Storage object at 0x7f1c15128080> archive = <swh.provenance.storage.archive.ArchiveStorage object at 0x7f1c15128160>
699 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.provenance.tests.test_origin_revision_layer::test_origin_revision_layer[local-db-with-path-with-merges-visits-01]
provenance = <swh.provenance.provenance.Provenance object at 0x7f1c151fd5f8> swh_storage = <swh.storage.postgresql.storage.Storage object at 0x7f1c151fdc88> archive = <swh.provenance.postgresql.archive.ArchivePostgreSQL object at 0x7f1c151fab70>
704 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.provenance.tests.test_origin_revision_layer::test_origin_revision_layer[local-db-without-path-with-merges-visits-01]
provenance = <swh.provenance.provenance.Provenance object at 0x7f1c15177710> swh_storage = <swh.storage.postgresql.storage.Storage object at 0x7f1c15177d68> archive = <swh.provenance.postgresql.archive.ArchivePostgreSQL object at 0x7f1c15036630>
774 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.provenance.tests.test_origin_revision_layer::test_origin_revision_layer[remote-archive-with-path-with-merges-visits-01]
provenance = <swh.provenance.provenance.Provenance object at 0x7f1c1510a550> swh_storage = <swh.storage.postgresql.storage.Storage object at 0x7f1c1510ac50> archive = <swh.provenance.storage.archive.ArchiveStorage object at 0x7f1c14fe79b0>
View Full Test Results (8 Failed · 323 Passed)

Event Timeline

aeviso created this revision.Jul 27 2021, 12:40 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 27 2021, 12:40 PM
swh-public-ci added a comment.Jul 27 2021, 12:46 PM

Build is green

Patch application report for D6026 (id=21785)

Could not rebase; Attempt merge onto 1ae32c0a61...

Updating 1ae32c0..1497b12
Fast-forward
 swh/provenance/graph.py                            |  19 +-
 swh/provenance/origin.py                           |  19 +-
 swh/provenance/postgresql/provenancedb_base.py     |   4 +-
 swh/provenance/tests/conftest.py                   | 128 +-------------
 swh/provenance/tests/data/README.md                |   2 +-
 swh/provenance/tests/data/generate_repo.py         |   2 +-
 .../data/history_graphs_with-merges_visits-01.yaml | 144 +++++++++++++++-
 .../data/origin-revision_with-merges_visits-01.txt |  67 +++++++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 7501 -> 10327 bytes
 swh/provenance/tests/test_origin_revision_layer.py | 192 +++++++++++++++++++++
 ...euristics.py => test_revision_content_layer.py} | 144 ++++++++++++++--
 11 files changed, 555 insertions(+), 166 deletions(-)
 create mode 100644 swh/provenance/tests/data/origin-revision_with-merges_visits-01.txt
 create mode 100644 swh/provenance/tests/test_origin_revision_layer.py
 rename swh/provenance/tests/{test_provenance_heuristics.py => test_revision_content_layer.py} (71%)
Changes applied before test
commit 1497b12545e3dd62125333e625d0893b75648f34
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jul 27 12:35:48 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit 509280132cb35df1432905097dafb24870690ee1
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 12:07:07 2021 +0200

    Simplify history graph creation and origin-revision algorithm

commit 505869054bda044c72d1a907631db552d173870f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 7 10:59:12 2021 +0200

    Add missing case to history graph yaml test file
    
    Fix typo too

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/267/ for more details.

Harbormaster completed remote builds in B22753: Diff 21785.Jul 27 2021, 12:46 PM
aeviso requested review of this revision.Jul 27 2021, 12:46 PM
douardda requested changes to this revision.Jul 27 2021, 6:11 PM
douardda added a subscriber: douardda.

I am not fond at all of the code duplication (between R-C and O-R synth file parsers), looks to me at least parts of it could be kept factorised in a dedicated module (I agree it should not live in conftest any more: too much code and logic now). It would then be best to have these test-helper functions tested themselves (as unitary as possible).

In any case this revision needs to be split at least in 2 revisions.

swh/provenance/postgresql/provenancedb_base.py
112

This is not related to adding a test, it's a fix of the business code. So either it should be in a dedicated revision or the git commit message should be updated to something like "Fix the origin_get() method [...]" and mention that a new test is added to validate the fix.

This revision now requires changes to proceed.Jul 27 2021, 6:11 PM
aeviso marked an inline comment as done.Jul 28 2021, 11:31 AM
aeviso updated this revision to Diff 21824.Jul 28 2021, 11:32 AM

rebase

swh-public-ci added a comment.Jul 28 2021, 11:38 AM

Build has FAILED

Patch application report for D6026 (id=21824)

Rebasing onto 509280132c...

Current branch diff-target is up to date.
Changes applied before test
commit f4ee11ace0d8a8648e9d8bec650bbf2292ed8454
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:15:03 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit 92237ff7db1d74aa42c62b5bff4575f2ee0f4087
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:11:49 2021 +0200

    Fix type conversion when retrieving origins' urls from PostgreSQL backend
    
    Correct some SQL code capitalization as well.

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/271/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/271/console

Harbormaster failed remote builds in B22788: Diff 21824!Jul 28 2021, 11:38 AM
aeviso updated this revision to Diff 21867.Jul 30 2021, 11:19 AM

rebase

swh-public-ci added a comment.Jul 30 2021, 11:26 AM

Build is green

Patch application report for D6026 (id=21867)

Rebasing onto 509280132c...

Current branch diff-target is up to date.
Changes applied before test
commit 16d91e21ff1429f897f953e122f6d21e2def0eb5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:15:03 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit 92237ff7db1d74aa42c62b5bff4575f2ee0f4087
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:11:49 2021 +0200

    Fix type conversion when retrieving origins' urls from PostgreSQL backend
    
    Correct some SQL code capitalization as well.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/278/ for more details.

Harbormaster completed remote builds in B22828: Diff 21867.Jul 30 2021, 11:26 AM
aeviso updated this revision to Diff 21880.Aug 2 2021, 1:56 PM

rebase

swh-public-ci added a comment.Aug 2 2021, 2:03 PM

Build is green

Patch application report for D6026 (id=21880)

Rebasing onto 509280132c...

Current branch diff-target is up to date.
Changes applied before test
commit 185d5b444625e5e603b8c9ae69a421398a950164
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 13:46:49 2021 +0200

    Add some missing type annotations and remove already done TODO's

commit 16d91e21ff1429f897f953e122f6d21e2def0eb5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:15:03 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit 92237ff7db1d74aa42c62b5bff4575f2ee0f4087
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:11:49 2021 +0200

    Fix type conversion when retrieving origins' urls from PostgreSQL backend
    
    Correct some SQL code capitalization as well.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/279/ for more details.

Harbormaster completed remote builds in B22843: Diff 21880.Aug 2 2021, 2:03 PM
aeviso updated this revision to Diff 21904.Aug 3 2021, 4:43 PM

rebase

aeviso added a child revision: D6053: Refactor the use of archive `Storage` object for testing.Aug 3 2021, 4:44 PM
swh-public-ci added a comment.Aug 3 2021, 4:50 PM

Build is green

Patch application report for D6026 (id=21904)

Rebasing onto 509280132c...

Current branch diff-target is up to date.
Changes applied before test
commit b738c4610cb84f3f5d10a16f667e4522f7d49377
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 13:46:49 2021 +0200

    Add some missing type annotations and remove already done TODO's

commit 16d91e21ff1429f897f953e122f6d21e2def0eb5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:15:03 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit 92237ff7db1d74aa42c62b5bff4575f2ee0f4087
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:11:49 2021 +0200

    Fix type conversion when retrieving origins' urls from PostgreSQL backend
    
    Correct some SQL code capitalization as well.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/280/ for more details.

Harbormaster completed remote builds in B22870: Diff 21904.Aug 3 2021, 4:50 PM
douardda accepted this revision.Aug 5 2021, 12:08 PM
This revision is now accepted and ready to land.Aug 5 2021, 12:08 PM
aeviso updated this revision to Diff 21960.Aug 6 2021, 3:17 PM

rebae

swh-public-ci added a comment.Aug 6 2021, 3:30 PM

Build is green

Patch application report for D6026 (id=21960)

Rebasing onto 3b145f15c2...

Current branch diff-target is up to date.
Changes applied before test
commit 91e4dcd4148da7763253f015e296dbc10ef2b564
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jul 30 13:46:49 2021 +0200

    Add some missing type annotations and remove already done TODO's

commit be098b32d8fb808d447d888c573e731952bc12b0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:15:03 2021 +0200

    Add test for origin-revision layer
    
    Rename revision-content layer test as well, for consistency.

commit c58d170f77661ddde409733462b7704705c5eefc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jul 28 11:11:49 2021 +0200

    Fix type conversion when retrieving origins' urls from PostgreSQL backend
    
    Correct some SQL code capitalization as well.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/286/ for more details.

Harbormaster completed remote builds in B22923: Diff 21960.Aug 6 2021, 3:30 PM
Closed by commit rDPROVc58d170f7766: Fix type conversion when retrieving origins' urls from PostgreSQL backend (authored by aeviso). · Explain WhyAug 6 2021, 4:00 PM
This revision was automatically updated to reflect the committed changes.
aeviso added a commit: rDPROVc58d170f7766: Fix type conversion when retrieving origins' urls from PostgreSQL backend.
aeviso added a commit: rDPROVbe098b32d8fb: Add test for origin-revision layer.
aeviso added a commit: rDPROV91e4dcd4148d: Add some missing type annotations and remove already done TODO's.

Revision Contents
Changeset List

Diff 21824

View Options

swh/provenance/postgresql/provenancedb_base.py

Loading...
View Options

swh/provenance/postgresql/provenancedb_with_path.py

Loading...
View Options

swh/provenance/postgresql/provenancedb_without_path.py

Loading...
View Options

swh/provenance/tests/conftest.py

Loading...
View Options

swh/provenance/tests/data/README.md

Loading...
View Options

swh/provenance/tests/data/with-merges.msgpack

Loading...
View Options

swh/provenance/tests/test_origin_revision_layer.py

Loading...
View Options

swh/provenance/tests/test_provenance_heuristics.py

Loading...
View Options

swh/provenance/tests/test_revision_content_layer.py

Loading...
About · Developer information · Legal