Paths

Table of Contentst

Differential D5388

Also test the provenance db with ArchiveStorage
ClosedPublic
Actions

Authored by douardda on Mar 30 2021, 5:30 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Commits

rDPROV5e89689ef08b: Also test the provenance db with ArchiveStorage

Summary

in addition to the ArchivePostgreSQL implementation.

Depends on D5387

Diff Detail

Repository

rDPROV Provenance database

Branch

wip

Lint

Lint Skipped

Unit

Unit Tests Skipped

Build Status

Buildable 20844
Build 32348: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 32347: arc lint + arc unit

Event Timeline

douardda created this revision.Mar 30 2021, 5:30 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptMar 30 2021, 5:30 PM

Build is green

Patch application report for D5388 (id=19290)

Could not rebase; Attempt merge onto 4a5a99ea7d...

Updating 4a5a99e..76bac66
Fast-forward
 swh/provenance/archive.py                          |   6 +-
 swh/provenance/cli.py                              |   2 +-
 swh/provenance/model.py                            |  83 ++++++++++----
 swh/provenance/origin.py                           |  53 +++------
 swh/provenance/postgresql/archive.py               |   6 +-
 swh/provenance/provenance.py                       |  62 ++++++----
 swh/provenance/revision.py                         |  44 +------
 swh/provenance/storage/archive.py                  |  10 +-
 swh/provenance/tests/conftest.py                   | 126 ++++++++++++++++++++-
 .../tests/data/synthetic_noroot_lower.txt          |  92 +++++++++++++++
 .../tests/data/synthetic_noroot_upper.txt          |  92 +++++++++++++++
 swh/provenance/tests/test_provenance_db.py         |  96 +++++++++++++---
 swh/provenance/tests/test_provenance_db_storage.py |  21 ++++
 13 files changed, 547 insertions(+), 146 deletions(-)
 create mode 100644 swh/provenance/tests/data/synthetic_noroot_lower.txt
 create mode 100644 swh/provenance/tests/data/synthetic_noroot_upper.txt
 create mode 100644 swh/provenance/tests/test_provenance_db_storage.py

Changes applied before test

commit 76bac6633c479ba1c98f8b485e1143e263739670
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Mar 30 15:24:21 2021 +0200

    Also test the provenance db with ArchiveStorage
    
    in addition to the ArchivePostgreSQL implementation.

commit 9583d0348f9013295f201944d63b69328fc87fd7
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Mar 30 14:49:38 2021 +0200

    Refactor the model
    
    - move all *Entry model classes in model.py,
    - kill the useless TreeEntry base class,
    - do not keep the archive instance as instance attribute of model
      objects, instead
    - implement "iterators" as explicit methods taking an ArchiveInterface
      object as argument,
    - refactor the code in provenance.py accordingly.

commit e97d2e2473efd39e08018914606438fd5e91b91d
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Mar 30 12:01:07 2021 +0200

    Simplify a bit origin.py
    
    - kill the useless OriginIterator class,
    - remove commented thread locks

commit 88c3910d2da58cc4167fb72b7ec530eec51cad8b
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Mar 25 15:01:35 2021 +0100

    Add a test for the (noroot, upper) case

commit 5d3f925e42530de891d94041046b2aec66d182c3
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 24 09:49:11 2021 +0100

    Add a test to compare the result of revision_add() with known results
    
    Use a 'synthetic' file to describe the expected behavior of the
    insertion of revisions in the provenance index.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/15/ for more details.

Harbormaster completed remote builds in B20358: Diff 19290.Mar 30 2021, 5:32 PM

douardda requested review of this revision.Mar 30 2021, 5:32 PM

douardda added a child revision: D5389: Improve tests.Mar 30 2021, 5:33 PM

vlorentz added a subscriber: vlorentz.Mar 30 2021, 5:39 PM

vlorentz added inline comments.

swh/provenance/tests/test_provenance_db_storage.py
10–14	why not `import *`?

ardumont accepted this revision.Mar 31 2021, 9:58 AM

ardumont added a subscriber: ardumont.

ardumont added inline comments.

swh/provenance/tests/test_provenance_db_storage.py
10–14	there are only 3 tests in there so far so my guess is it was to avoid the noqa (but then david may have forgotten to remove it)

This revision is now accepted and ready to land.Mar 31 2021, 9:58 AM

douardda added inline comments.Mar 31 2021, 3:19 PM

swh/provenance/tests/test_provenance_db_storage.py
10–14	in both cases the noqa is mandatory (otherwise flake8 complains about unused imported stuff)

rebase

Build is green

Patch application report for D5388 (id=19835)

Could not rebase; Attempt merge onto 62617e5006...

Updating 62617e5..5e89689
Fast-forward
 swh/provenance/archive.py                          |   6 +-
 swh/provenance/cli.py                              |   2 +-
 swh/provenance/model.py                            |  83 ++++++++++----
 swh/provenance/origin.py                           |  53 +++------
 swh/provenance/postgresql/archive.py               |   6 +-
 swh/provenance/provenance.py                       |  62 ++++++----
 swh/provenance/revision.py                         |  44 +------
 swh/provenance/storage/archive.py                  |  10 +-
 swh/provenance/tests/conftest.py                   | 126 ++++++++++++++++++++-
 .../tests/data/synthetic_noroot_lower.txt          |  92 +++++++++++++++
 .../tests/data/synthetic_noroot_upper.txt          |  92 +++++++++++++++
 swh/provenance/tests/test_provenance_db.py         |  96 +++++++++++++---
 swh/provenance/tests/test_provenance_db_storage.py |  21 ++++
 13 files changed, 547 insertions(+), 146 deletions(-)
 create mode 100644 swh/provenance/tests/data/synthetic_noroot_lower.txt
 create mode 100644 swh/provenance/tests/data/synthetic_noroot_upper.txt
 create mode 100644 swh/provenance/tests/test_provenance_db_storage.py

Changes applied before test

commit 5e89689ef08b4a437eebca4d2a32834dfa4740dc
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Mar 30 15:24:21 2021 +0200

    Also test the provenance db with ArchiveStorage
    
    in addition to the ArchivePostgreSQL implementation.

commit a23a33c5a77da059f72835c0400c4384a04b7f64
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Mar 30 14:49:38 2021 +0200

    Refactor the model
    
    - move all *Entry model classes in model.py,
    - kill the useless TreeEntry base class,
    - do not keep the archive instance as instance attribute of model
      objects, instead
    - implement "iterators" as explicit methods taking an ArchiveInterface
      object as argument,
    - refactor the code in provenance.py accordingly.

commit 7eaeebb6091a13105ddee289fdcf9f70cb367c8e
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Mar 30 12:01:07 2021 +0200

    Simplify a bit origin.py
    
    - kill the useless OriginIterator class,
    - remove commented thread locks

commit 8853314af9817988a58450937c9d25097e66e741
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Mar 25 15:01:35 2021 +0100

    Add a test for the (noroot, upper) case

commit adbc99dd357dc452081df1965805756987b7e09b
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 24 09:49:11 2021 +0100

    Add a test to compare the result of revision_add() with known results
    
    Use a 'synthetic' file to describe the expected behavior of the
    insertion of revisions in the provenance index.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/19/ for more details.

Harbormaster completed remote builds in B20844: Diff 19835.Apr 19 2021, 4:45 PM

Closed by commit rDPROV5e89689ef08b: Also test the provenance db with ArchiveStorage (authored by douardda). · Explain WhyApr 19 2021, 4:56 PM

This revision was automatically updated to reflect the committed changes.

douardda added a commit: rDPROV5e89689ef08b: Also test the provenance db with ArchiveStorage.