Page MenuHomeSoftware Heritage

Also test the provenance db with ArchiveStorage
ClosedPublic

Authored by douardda on Mar 30 2021, 5:30 PM.

Details

Summary

in addition to the ArchivePostgreSQL implementation.

Depends on D5387

Event Timeline

Build is green

Patch application report for D5388 (id=19290)

Could not rebase; Attempt merge onto 4a5a99ea7d...

Updating 4a5a99e..76bac66
Fast-forward
 swh/provenance/archive.py                          |   6 +-
 swh/provenance/cli.py                              |   2 +-
 swh/provenance/model.py                            |  83 ++++++++++----
 swh/provenance/origin.py                           |  53 +++------
 swh/provenance/postgresql/archive.py               |   6 +-
 swh/provenance/provenance.py                       |  62 ++++++----
 swh/provenance/revision.py                         |  44 +------
 swh/provenance/storage/archive.py                  |  10 +-
 swh/provenance/tests/conftest.py                   | 126 ++++++++++++++++++++-
 .../tests/data/synthetic_noroot_lower.txt          |  92 +++++++++++++++
 .../tests/data/synthetic_noroot_upper.txt          |  92 +++++++++++++++
 swh/provenance/tests/test_provenance_db.py         |  96 +++++++++++++---
 swh/provenance/tests/test_provenance_db_storage.py |  21 ++++
 13 files changed, 547 insertions(+), 146 deletions(-)
 create mode 100644 swh/provenance/tests/data/synthetic_noroot_lower.txt
 create mode 100644 swh/provenance/tests/data/synthetic_noroot_upper.txt
 create mode 100644 swh/provenance/tests/test_provenance_db_storage.py
Changes applied before test
commit 76bac6633c479ba1c98f8b485e1143e263739670
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Mar 30 15:24:21 2021 +0200

    Also test the provenance db with ArchiveStorage
    
    in addition to the ArchivePostgreSQL implementation.

commit 9583d0348f9013295f201944d63b69328fc87fd7
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Mar 30 14:49:38 2021 +0200

    Refactor the model
    
    - move all *Entry model classes in model.py,
    - kill the useless TreeEntry base class,
    - do not keep the archive instance as instance attribute of model
      objects, instead
    - implement "iterators" as explicit methods taking an ArchiveInterface
      object as argument,
    - refactor the code in provenance.py accordingly.

commit e97d2e2473efd39e08018914606438fd5e91b91d
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Mar 30 12:01:07 2021 +0200

    Simplify a bit origin.py
    
    - kill the useless OriginIterator class,
    - remove commented thread locks

commit 88c3910d2da58cc4167fb72b7ec530eec51cad8b
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Mar 25 15:01:35 2021 +0100

    Add a test for the (noroot, upper) case

commit 5d3f925e42530de891d94041046b2aec66d182c3
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 24 09:49:11 2021 +0100

    Add a test to compare the result of revision_add() with known results
    
    Use a 'synthetic' file to describe the expected behavior of the
    insertion of revisions in the provenance index.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/15/ for more details.

vlorentz added inline comments.
swh/provenance/tests/test_provenance_db_storage.py
10–14

why not import *?

ardumont added a subscriber: ardumont.
ardumont added inline comments.
swh/provenance/tests/test_provenance_db_storage.py
10–14

there are only 3 tests in there so far so my guess is it was to avoid the noqa (but then david may have forgotten to remove it)

This revision is now accepted and ready to land.Mar 31 2021, 9:58 AM
swh/provenance/tests/test_provenance_db_storage.py
10–14

in both cases the noqa is mandatory (otherwise flake8 complains about unused imported stuff)

Build is green

Patch application report for D5388 (id=19835)

Could not rebase; Attempt merge onto 62617e5006...

Updating 62617e5..5e89689
Fast-forward
 swh/provenance/archive.py                          |   6 +-
 swh/provenance/cli.py                              |   2 +-
 swh/provenance/model.py                            |  83 ++++++++++----
 swh/provenance/origin.py                           |  53 +++------
 swh/provenance/postgresql/archive.py               |   6 +-
 swh/provenance/provenance.py                       |  62 ++++++----
 swh/provenance/revision.py                         |  44 +------
 swh/provenance/storage/archive.py                  |  10 +-
 swh/provenance/tests/conftest.py                   | 126 ++++++++++++++++++++-
 .../tests/data/synthetic_noroot_lower.txt          |  92 +++++++++++++++
 .../tests/data/synthetic_noroot_upper.txt          |  92 +++++++++++++++
 swh/provenance/tests/test_provenance_db.py         |  96 +++++++++++++---
 swh/provenance/tests/test_provenance_db_storage.py |  21 ++++
 13 files changed, 547 insertions(+), 146 deletions(-)
 create mode 100644 swh/provenance/tests/data/synthetic_noroot_lower.txt
 create mode 100644 swh/provenance/tests/data/synthetic_noroot_upper.txt
 create mode 100644 swh/provenance/tests/test_provenance_db_storage.py
Changes applied before test
commit 5e89689ef08b4a437eebca4d2a32834dfa4740dc
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Mar 30 15:24:21 2021 +0200

    Also test the provenance db with ArchiveStorage
    
    in addition to the ArchivePostgreSQL implementation.

commit a23a33c5a77da059f72835c0400c4384a04b7f64
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Mar 30 14:49:38 2021 +0200

    Refactor the model
    
    - move all *Entry model classes in model.py,
    - kill the useless TreeEntry base class,
    - do not keep the archive instance as instance attribute of model
      objects, instead
    - implement "iterators" as explicit methods taking an ArchiveInterface
      object as argument,
    - refactor the code in provenance.py accordingly.

commit 7eaeebb6091a13105ddee289fdcf9f70cb367c8e
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Mar 30 12:01:07 2021 +0200

    Simplify a bit origin.py
    
    - kill the useless OriginIterator class,
    - remove commented thread locks

commit 8853314af9817988a58450937c9d25097e66e741
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Mar 25 15:01:35 2021 +0100

    Add a test for the (noroot, upper) case

commit adbc99dd357dc452081df1965805756987b7e09b
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 24 09:49:11 2021 +0100

    Add a test to compare the result of revision_add() with known results
    
    Use a 'synthetic' file to describe the expected behavior of the
    insertion of revisions in the provenance index.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/19/ for more details.