Exhaustively added type annotations to all test functions (there is one pending
TODO though) and applied the necessary fixes for the code to be compliant.
Depends on D5961.
Details
Details
- Reviewers
vlorentz - Group Reviewers
Reviewers - Commits
- rDPROV2e46f7c34b0b: Improve type annotations for test
Diff Detail
Diff Detail
- Repository
- rDPROV Provenance database
- Branch
- master
- Lint
Lint Skipped - Unit
Unit Tests Skipped - Build Status
Buildable 22402 Build 34896: Phabricator diff pipeline on jenkins Jenkins console · Jenkins Build 34895: arc lint + arc unit
Event Timeline
Comment Actions
Build is green
Patch application report for D5962 (id=21416)
Could not rebase; Attempt merge onto d892b29e40...
Updating d892b29..68d5316
Fast-forward
swh/provenance/__init__.py | 57 +--
swh/provenance/archive.py | 25 +-
swh/provenance/backend.py | 357 ++++++++++++++++++
swh/provenance/cli.py | 79 ++--
swh/provenance/graph.py | 34 +-
swh/provenance/model.py | 91 +++--
swh/provenance/origin.py | 33 +-
swh/provenance/postgresql/archive.py | 130 +++----
swh/provenance/postgresql/db_utils.py | 61 ----
swh/provenance/postgresql/provenancedb_base.py | 404 ++++++++++++---------
.../postgresql/provenancedb_with_path.py | 117 +++---
.../postgresql/provenancedb_without_path.py | 96 ++---
swh/provenance/provenance.py | 387 ++++++++++----------
swh/provenance/revision.py | 46 +--
swh/provenance/sql/30-schema.sql | 30 +-
swh/provenance/storage/archive.py | 39 +-
swh/provenance/tests/conftest.py | 69 ++--
.../tests/data/generate_storage_from_git.py | 3 +-
.../data/history_graphs_with-merges_visits-01.yaml | 55 +++
swh/provenance/tests/data/with-merges.msgpack | Bin 0 -> 7501 bytes
...repo_with_merges.yaml => with-merges_repo.yaml} | 0
...s-visits-01.yaml => with-merges_visits-01.yaml} | 0
swh/provenance/tests/test_archive_interface.py | 51 +++
swh/provenance/tests/test_cli.py | 16 +-
swh/provenance/tests/test_conftest.py | 9 +-
swh/provenance/tests/test_history_graph.py | 74 ++++
swh/provenance/tests/test_isochrone_graph.py | 17 +-
swh/provenance/tests/test_origin_iterator.py | 23 +-
swh/provenance/tests/test_provenance_db.py | 29 +-
swh/provenance/tests/test_provenance_heuristics.py | 120 +++---
swh/provenance/tests/test_revision_iterator.py | 6 +-
31 files changed, 1503 insertions(+), 955 deletions(-)
create mode 100644 swh/provenance/backend.py
delete mode 100644 swh/provenance/postgresql/db_utils.py
create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
create mode 100644 swh/provenance/tests/data/with-merges.msgpack
rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
create mode 100644 swh/provenance/tests/test_archive_interface.py
create mode 100644 swh/provenance/tests/test_history_graph.pyChanges applied before test
commit 68d53169d8f3b4e88059c078bd23972724df8753
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Jul 2 01:05:24 2021 +0200
Improve type annotations for test
Exhaustively added type annotations to all test functions (there is one pending
TODO though) and applied the necessary fixes for the code to be compliant.
commit e48a48f1760247f0bb9c7f9e85ab8dfc3e8b3895
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 22:55:51 2021 +0200
Improve type annotations
All main module's files were updated with exhaustive type annotations,
making the necessary (minor) fixes for the code to be compliant with it.
Also, `__future__. Annotations` was imported to avoid using strings when
possible.
commit 177dd6488e3255e65e95b6fddee56978036d3e54
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 17:48:12 2021 +0200
Improve typing annotations for `origin` and `revision` modules
This implie fixing `RevisionCSVIterator`, and updating cli and related tests.
commit 4f0da58cde268c8c378f9866d9325d7f3a98bd8c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 17:34:08 2021 +0200
Remove `db_utils` in favour of `swh.core.db.BaseDb` methods
This allows to remove some unnecessary `psycopg2` dependencies in modules that
don't deal directly with SQL queries.
commit 0c3c7817cfd267bf3b77ce2e594ea7feb3ec96d5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Jun 29 14:28:54 2021 +0200
Force `snapshot_get_heads` to return revisions in chronological order
commit afb67f665ab00c03c0ca33e96b1bfc109c827c58
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 19:21:58 2021 +0200
Add `ProvenanceStorageInterface` as discussed during backend design
Rework backend-related classes to properly use the new interface.
Adapt tests to the new structure as well.
commit 3672235c3258cf93fb37a82d060bf40ba1761b8b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 14:37:50 2021 +0200
Move `ProvenanceBackend` implementation to a separate file
commit 6f4da6fed7e663273627ad4a46c8489ef0a0e784
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 13:47:26 2021 +0200
Use `RealDictCursor` in `ProvenanceDBBase`
to improve the way `ProvenanceResult`s are generated.
commit 07a30e43a76e170ab03764035da68dcf7db1fc3b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 14:28:32 2021 +0200
Rework `ProvenanceInterface` as discussed during backend design
Add `ProvenanceResult` class to be returned by `content_find_first` and
`content_find_all` methods. Rename some methods. Improve type annotations.
commit 2fd3f56b57f8db6691ae6b8b7cb7ac557b764172
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:31:16 2021 +0200
Add tests for history graph topology
commit d45d6ff9e9317ecfe38d584df7297c548b654d28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:10:38 2021 +0200
Fix database queries related to the origin-revision layer
This required allowing null dates in the `revision` table so that revision can be added
by the origin-revision layer algorithm but not recognized as already processed by the
revision-content layer. Revision and origin entries are now inserted in the database
prior to inserting rows to revision_in_origin and revision_before_revision relations,
so that internal ids are properly resolved.
commit 0e2a3c64ce3c368b53c101c541e8aebcde789477
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Jun 25 13:38:26 2021 +0200
Add test to compare both `ArchiveInterface` implementations
Improve documentation of the interface and complete pending TODO's.
commit 98bba93cccece2b47ec4cd5887997cb5bede1e87
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:25:15 2021 +0200
Rename test files to keep naming convension
Also added missing .msgpack file dump for new with-merges repository.
commit fa9198afb71bcf3b8abea07d88d763a430f7358e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:05:24 2021 +0200
Refactor `ArchiveInterface` to fit origin-revision layer needs
Replace `revision_get` method by `revision_get_parents` returning an iterable of
parents' ids only, instead of a swh.model.model.Revision object.
commit 9e0c1aa099073887206c9334e17b49ee31bbef9a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Jun 23 20:00:40 2021 +0200
Use `Sha1Git` type to explicitly state the kind of identifiers
Previous occurrences of `bytes` and `Sha1` are now correctly using `Sha1Git`.
Also, some bytes conversion methods were replaced by their counterparts in
the swh.model.hashutil module.
commit a27ffff67b6b14bf37d153bb9b1d1c2ae63773fc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Jun 23 19:12:06 2021 +0200
Add support for sha1 identifiers for originsSee https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/226/ for more details.
Comment Actions
Build is green
Patch application report for D5962 (id=21423)
Could not rebase; Attempt merge onto d892b29e40...
Updating d892b29..bfd978b
Fast-forward
swh/provenance/__init__.py | 98 +++--
swh/provenance/archive.py | 25 +-
swh/provenance/backend.py | 357 ++++++++++++++++++
swh/provenance/cli.py | 79 ++--
swh/provenance/graph.py | 34 +-
swh/provenance/model.py | 91 +++--
swh/provenance/origin.py | 33 +-
swh/provenance/postgresql/archive.py | 130 +++----
swh/provenance/postgresql/db_utils.py | 61 ----
swh/provenance/postgresql/provenancedb_base.py | 404 ++++++++++++---------
.../postgresql/provenancedb_with_path.py | 117 +++---
.../postgresql/provenancedb_without_path.py | 96 ++---
swh/provenance/provenance.py | 387 ++++++++++----------
swh/provenance/revision.py | 46 +--
swh/provenance/sql/30-schema.sql | 30 +-
swh/provenance/storage/archive.py | 39 +-
swh/provenance/tests/conftest.py | 69 ++--
.../tests/data/generate_storage_from_git.py | 3 +-
.../data/history_graphs_with-merges_visits-01.yaml | 55 +++
swh/provenance/tests/data/with-merges.msgpack | Bin 0 -> 7501 bytes
...repo_with_merges.yaml => with-merges_repo.yaml} | 0
...s-visits-01.yaml => with-merges_visits-01.yaml} | 0
swh/provenance/tests/test_archive_interface.py | 51 +++
swh/provenance/tests/test_cli.py | 16 +-
swh/provenance/tests/test_conftest.py | 9 +-
swh/provenance/tests/test_history_graph.py | 74 ++++
swh/provenance/tests/test_isochrone_graph.py | 17 +-
swh/provenance/tests/test_origin_iterator.py | 23 +-
swh/provenance/tests/test_provenance_db.py | 29 +-
swh/provenance/tests/test_provenance_heuristics.py | 120 +++---
swh/provenance/tests/test_revision_iterator.py | 6 +-
31 files changed, 1541 insertions(+), 958 deletions(-)
create mode 100644 swh/provenance/backend.py
delete mode 100644 swh/provenance/postgresql/db_utils.py
create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
create mode 100644 swh/provenance/tests/data/with-merges.msgpack
rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
create mode 100644 swh/provenance/tests/test_archive_interface.py
create mode 100644 swh/provenance/tests/test_history_graph.pyChanges applied before test
commit bfd978b6f0289a18a805bc5d7740fab21df4ede1
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Jul 2 01:05:24 2021 +0200
Improve type annotations for test
Exhaustively added type annotations to all test functions (there is one pending
TODO though) and applied the necessary fixes for the code to be compliant.
commit 70958e6d161160736ea89faff57bb262ca764bd9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 22:55:51 2021 +0200
Improve type annotations throughout the module
All main module's files were updated with exhaustive type annotations,
making the necessary (minor) fixes for the code to be compliant with it.
Also, `__future__. annotations` was imported to avoid using strings when
possible.
commit 177dd6488e3255e65e95b6fddee56978036d3e54
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 17:48:12 2021 +0200
Improve typing annotations for `origin` and `revision` modules
This implie fixing `RevisionCSVIterator`, and updating cli and related tests.
commit 4f0da58cde268c8c378f9866d9325d7f3a98bd8c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 17:34:08 2021 +0200
Remove `db_utils` in favour of `swh.core.db.BaseDb` methods
This allows to remove some unnecessary `psycopg2` dependencies in modules that
don't deal directly with SQL queries.
commit 0c3c7817cfd267bf3b77ce2e594ea7feb3ec96d5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Jun 29 14:28:54 2021 +0200
Force `snapshot_get_heads` to return revisions in chronological order
commit afb67f665ab00c03c0ca33e96b1bfc109c827c58
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 19:21:58 2021 +0200
Add `ProvenanceStorageInterface` as discussed during backend design
Rework backend-related classes to properly use the new interface.
Adapt tests to the new structure as well.
commit 3672235c3258cf93fb37a82d060bf40ba1761b8b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 14:37:50 2021 +0200
Move `ProvenanceBackend` implementation to a separate file
commit 6f4da6fed7e663273627ad4a46c8489ef0a0e784
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 13:47:26 2021 +0200
Use `RealDictCursor` in `ProvenanceDBBase`
to improve the way `ProvenanceResult`s are generated.
commit 07a30e43a76e170ab03764035da68dcf7db1fc3b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 14:28:32 2021 +0200
Rework `ProvenanceInterface` as discussed during backend design
Add `ProvenanceResult` class to be returned by `content_find_first` and
`content_find_all` methods. Rename some methods. Improve type annotations.
commit 2fd3f56b57f8db6691ae6b8b7cb7ac557b764172
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:31:16 2021 +0200
Add tests for history graph topology
commit d45d6ff9e9317ecfe38d584df7297c548b654d28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:10:38 2021 +0200
Fix database queries related to the origin-revision layer
This required allowing null dates in the `revision` table so that revision can be added
by the origin-revision layer algorithm but not recognized as already processed by the
revision-content layer. Revision and origin entries are now inserted in the database
prior to inserting rows to revision_in_origin and revision_before_revision relations,
so that internal ids are properly resolved.
commit 0e2a3c64ce3c368b53c101c541e8aebcde789477
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Jun 25 13:38:26 2021 +0200
Add test to compare both `ArchiveInterface` implementations
Improve documentation of the interface and complete pending TODO's.
commit 98bba93cccece2b47ec4cd5887997cb5bede1e87
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:25:15 2021 +0200
Rename test files to keep naming convension
Also added missing .msgpack file dump for new with-merges repository.
commit fa9198afb71bcf3b8abea07d88d763a430f7358e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:05:24 2021 +0200
Refactor `ArchiveInterface` to fit origin-revision layer needs
Replace `revision_get` method by `revision_get_parents` returning an iterable of
parents' ids only, instead of a swh.model.model.Revision object.
commit 9e0c1aa099073887206c9334e17b49ee31bbef9a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Jun 23 20:00:40 2021 +0200
Use `Sha1Git` type to explicitly state the kind of identifiers
Previous occurrences of `bytes` and `Sha1` are now correctly using `Sha1Git`.
Also, some bytes conversion methods were replaced by their counterparts in
the swh.model.hashutil module.
commit a27ffff67b6b14bf37d153bb9b1d1c2ae63773fc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Jun 23 19:12:06 2021 +0200
Add support for sha1 identifiers for originsSee https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/228/ for more details.
Comment Actions
Build is green
Patch application report for D5962 (id=21424)
Could not rebase; Attempt merge onto d892b29e40...
Updating d892b29..c0a09de
Fast-forward
swh/provenance/__init__.py | 98 +++--
swh/provenance/archive.py | 25 +-
swh/provenance/backend.py | 357 ++++++++++++++++++
swh/provenance/cli.py | 79 ++--
swh/provenance/graph.py | 34 +-
swh/provenance/model.py | 91 +++--
swh/provenance/origin.py | 33 +-
swh/provenance/postgresql/archive.py | 130 +++----
swh/provenance/postgresql/db_utils.py | 61 ----
swh/provenance/postgresql/provenancedb_base.py | 404 ++++++++++++---------
.../postgresql/provenancedb_with_path.py | 117 +++---
.../postgresql/provenancedb_without_path.py | 96 ++---
swh/provenance/provenance.py | 387 ++++++++++----------
swh/provenance/revision.py | 46 +--
swh/provenance/sql/30-schema.sql | 30 +-
swh/provenance/storage/archive.py | 39 +-
swh/provenance/tests/conftest.py | 70 ++--
.../tests/data/generate_storage_from_git.py | 3 +-
.../data/history_graphs_with-merges_visits-01.yaml | 55 +++
swh/provenance/tests/data/with-merges.msgpack | Bin 0 -> 7501 bytes
...repo_with_merges.yaml => with-merges_repo.yaml} | 0
...s-visits-01.yaml => with-merges_visits-01.yaml} | 0
swh/provenance/tests/test_archive_interface.py | 51 +++
swh/provenance/tests/test_cli.py | 16 +-
swh/provenance/tests/test_conftest.py | 9 +-
swh/provenance/tests/test_history_graph.py | 74 ++++
swh/provenance/tests/test_isochrone_graph.py | 17 +-
swh/provenance/tests/test_origin_iterator.py | 23 +-
swh/provenance/tests/test_provenance_db.py | 29 +-
swh/provenance/tests/test_provenance_heuristics.py | 120 +++---
swh/provenance/tests/test_revision_iterator.py | 6 +-
31 files changed, 1542 insertions(+), 958 deletions(-)
create mode 100644 swh/provenance/backend.py
delete mode 100644 swh/provenance/postgresql/db_utils.py
create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
create mode 100644 swh/provenance/tests/data/with-merges.msgpack
rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
create mode 100644 swh/provenance/tests/test_archive_interface.py
create mode 100644 swh/provenance/tests/test_history_graph.pyChanges applied before test
commit c0a09de468386f74bc6db05c1ddbe21fb125ec50
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Jul 2 01:05:24 2021 +0200
Improve type annotations for test
Exhaustively added type annotations to all test functions (there are some pending
TODO though) and applied the necessary fixes for the code to be compliant.
commit 70958e6d161160736ea89faff57bb262ca764bd9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 22:55:51 2021 +0200
Improve type annotations throughout the module
All main module's files were updated with exhaustive type annotations,
making the necessary (minor) fixes for the code to be compliant with it.
Also, `__future__. annotations` was imported to avoid using strings when
possible.
commit 177dd6488e3255e65e95b6fddee56978036d3e54
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 17:48:12 2021 +0200
Improve typing annotations for `origin` and `revision` modules
This implie fixing `RevisionCSVIterator`, and updating cli and related tests.
commit 4f0da58cde268c8c378f9866d9325d7f3a98bd8c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 17:34:08 2021 +0200
Remove `db_utils` in favour of `swh.core.db.BaseDb` methods
This allows to remove some unnecessary `psycopg2` dependencies in modules that
don't deal directly with SQL queries.
commit 0c3c7817cfd267bf3b77ce2e594ea7feb3ec96d5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Jun 29 14:28:54 2021 +0200
Force `snapshot_get_heads` to return revisions in chronological order
commit afb67f665ab00c03c0ca33e96b1bfc109c827c58
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 19:21:58 2021 +0200
Add `ProvenanceStorageInterface` as discussed during backend design
Rework backend-related classes to properly use the new interface.
Adapt tests to the new structure as well.
commit 3672235c3258cf93fb37a82d060bf40ba1761b8b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 14:37:50 2021 +0200
Move `ProvenanceBackend` implementation to a separate file
commit 6f4da6fed7e663273627ad4a46c8489ef0a0e784
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 13:47:26 2021 +0200
Use `RealDictCursor` in `ProvenanceDBBase`
to improve the way `ProvenanceResult`s are generated.
commit 07a30e43a76e170ab03764035da68dcf7db1fc3b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 14:28:32 2021 +0200
Rework `ProvenanceInterface` as discussed during backend design
Add `ProvenanceResult` class to be returned by `content_find_first` and
`content_find_all` methods. Rename some methods. Improve type annotations.
commit 2fd3f56b57f8db6691ae6b8b7cb7ac557b764172
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:31:16 2021 +0200
Add tests for history graph topology
commit d45d6ff9e9317ecfe38d584df7297c548b654d28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:10:38 2021 +0200
Fix database queries related to the origin-revision layer
This required allowing null dates in the `revision` table so that revision can be added
by the origin-revision layer algorithm but not recognized as already processed by the
revision-content layer. Revision and origin entries are now inserted in the database
prior to inserting rows to revision_in_origin and revision_before_revision relations,
so that internal ids are properly resolved.
commit 0e2a3c64ce3c368b53c101c541e8aebcde789477
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Jun 25 13:38:26 2021 +0200
Add test to compare both `ArchiveInterface` implementations
Improve documentation of the interface and complete pending TODO's.
commit 98bba93cccece2b47ec4cd5887997cb5bede1e87
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:25:15 2021 +0200
Rename test files to keep naming convension
Also added missing .msgpack file dump for new with-merges repository.
commit fa9198afb71bcf3b8abea07d88d763a430f7358e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:05:24 2021 +0200
Refactor `ArchiveInterface` to fit origin-revision layer needs
Replace `revision_get` method by `revision_get_parents` returning an iterable of
parents' ids only, instead of a swh.model.model.Revision object.
commit 9e0c1aa099073887206c9334e17b49ee31bbef9a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Jun 23 20:00:40 2021 +0200
Use `Sha1Git` type to explicitly state the kind of identifiers
Previous occurrences of `bytes` and `Sha1` are now correctly using `Sha1Git`.
Also, some bytes conversion methods were replaced by their counterparts in
the swh.model.hashutil module.
commit a27ffff67b6b14bf37d153bb9b1d1c2ae63773fc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Jun 23 19:12:06 2021 +0200
Add support for sha1 identifiers for originsSee https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/229/ for more details.
Comment Actions
Build is green
Patch application report for D5962 (id=21431)
Could not rebase; Attempt merge onto d892b29e40...
Updating d892b29..a47192e
Fast-forward
swh/provenance/__init__.py | 98 +++--
swh/provenance/archive.py | 25 +-
swh/provenance/backend.py | 322 ++++++++++++++++
swh/provenance/cli.py | 79 ++--
swh/provenance/graph.py | 34 +-
swh/provenance/model.py | 91 +++--
swh/provenance/origin.py | 33 +-
swh/provenance/postgresql/archive.py | 130 +++----
swh/provenance/postgresql/db_utils.py | 61 ----
swh/provenance/postgresql/provenancedb_base.py | 404 ++++++++++++---------
.../postgresql/provenancedb_with_path.py | 117 +++---
.../postgresql/provenancedb_without_path.py | 96 ++---
swh/provenance/provenance.py | 387 ++++++++++----------
swh/provenance/revision.py | 46 +--
swh/provenance/sql/30-schema.sql | 30 +-
swh/provenance/storage/archive.py | 39 +-
swh/provenance/tests/conftest.py | 70 ++--
.../tests/data/generate_storage_from_git.py | 3 +-
.../data/history_graphs_with-merges_visits-01.yaml | 55 +++
swh/provenance/tests/data/with-merges.msgpack | Bin 0 -> 7501 bytes
...repo_with_merges.yaml => with-merges_repo.yaml} | 0
...s-visits-01.yaml => with-merges_visits-01.yaml} | 0
swh/provenance/tests/test_archive_interface.py | 51 +++
swh/provenance/tests/test_cli.py | 16 +-
swh/provenance/tests/test_conftest.py | 9 +-
swh/provenance/tests/test_history_graph.py | 74 ++++
swh/provenance/tests/test_isochrone_graph.py | 17 +-
swh/provenance/tests/test_origin_iterator.py | 23 +-
swh/provenance/tests/test_provenance_db.py | 29 +-
swh/provenance/tests/test_provenance_heuristics.py | 120 +++---
swh/provenance/tests/test_revision_iterator.py | 6 +-
31 files changed, 1507 insertions(+), 958 deletions(-)
create mode 100644 swh/provenance/backend.py
delete mode 100644 swh/provenance/postgresql/db_utils.py
create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
create mode 100644 swh/provenance/tests/data/with-merges.msgpack
rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
create mode 100644 swh/provenance/tests/test_archive_interface.py
create mode 100644 swh/provenance/tests/test_history_graph.pyChanges applied before test
commit a47192edc23ab22601311fc81805d0064b25cf82
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Jul 2 01:05:24 2021 +0200
Improve type annotations for test
Exhaustively added type annotations to all test functions (there are some pending
TODO though) and applied the necessary fixes for the code to be compliant.
commit 23ac7d3a328599517caa03fa0a2242fc477d2dd5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 22:55:51 2021 +0200
Improve type annotations throughout the module
All main module's files were updated with exhaustive type annotations,
making the necessary (minor) fixes for the code to be compliant with it.
Also, `__future__. annotations` was imported to avoid using strings when
possible.
commit ffa4cb98c6aa99fcc2e43c8dea0543c2b2379d8b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 17:48:12 2021 +0200
Improve typing annotations for `origin` and `revision` modules
This implie fixing `RevisionCSVIterator`, and updating cli and related tests.
commit a4eb765f92d25d45ddea178aaeebc375d69e3830
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 17:34:08 2021 +0200
Remove `db_utils` in favour of `swh.core.db.BaseDb` methods
This allows to remove some unnecessary `psycopg2` dependencies in modules that
don't deal directly with SQL queries.
commit 2a1aeb023627ed31ee4eea617790607e1fa1ba68
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Jun 29 14:28:54 2021 +0200
Force `snapshot_get_heads` to return revisions in chronological order
commit f819e4332df40b1ef35ff737f2558de570379473
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 19:21:58 2021 +0200
Add `ProvenanceStorageInterface` as discussed during backend design
Rework backend-related classes to properly use the new interface.
Adapt tests to the new structure as well.
commit 3672235c3258cf93fb37a82d060bf40ba1761b8b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 14:37:50 2021 +0200
Move `ProvenanceBackend` implementation to a separate file
commit 6f4da6fed7e663273627ad4a46c8489ef0a0e784
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 13:47:26 2021 +0200
Use `RealDictCursor` in `ProvenanceDBBase`
to improve the way `ProvenanceResult`s are generated.
commit 07a30e43a76e170ab03764035da68dcf7db1fc3b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 14:28:32 2021 +0200
Rework `ProvenanceInterface` as discussed during backend design
Add `ProvenanceResult` class to be returned by `content_find_first` and
`content_find_all` methods. Rename some methods. Improve type annotations.
commit 2fd3f56b57f8db6691ae6b8b7cb7ac557b764172
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:31:16 2021 +0200
Add tests for history graph topology
commit d45d6ff9e9317ecfe38d584df7297c548b654d28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:10:38 2021 +0200
Fix database queries related to the origin-revision layer
This required allowing null dates in the `revision` table so that revision can be added
by the origin-revision layer algorithm but not recognized as already processed by the
revision-content layer. Revision and origin entries are now inserted in the database
prior to inserting rows to revision_in_origin and revision_before_revision relations,
so that internal ids are properly resolved.
commit 0e2a3c64ce3c368b53c101c541e8aebcde789477
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Jun 25 13:38:26 2021 +0200
Add test to compare both `ArchiveInterface` implementations
Improve documentation of the interface and complete pending TODO's.
commit 98bba93cccece2b47ec4cd5887997cb5bede1e87
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:25:15 2021 +0200
Rename test files to keep naming convension
Also added missing .msgpack file dump for new with-merges repository.
commit fa9198afb71bcf3b8abea07d88d763a430f7358e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:05:24 2021 +0200
Refactor `ArchiveInterface` to fit origin-revision layer needs
Replace `revision_get` method by `revision_get_parents` returning an iterable of
parents' ids only, instead of a swh.model.model.Revision object.
commit 9e0c1aa099073887206c9334e17b49ee31bbef9a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Jun 23 20:00:40 2021 +0200
Use `Sha1Git` type to explicitly state the kind of identifiers
Previous occurrences of `bytes` and `Sha1` are now correctly using `Sha1Git`.
Also, some bytes conversion methods were replaced by their counterparts in
the swh.model.hashutil module.
commit a27ffff67b6b14bf37d153bb9b1d1c2ae63773fc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Jun 23 19:12:06 2021 +0200
Add support for sha1 identifiers for originsSee https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/235/ for more details.
Comment Actions
Build is green
Patch application report for D5962 (id=21438)
Could not rebase; Attempt merge onto d892b29e40...
Updating d892b29..b9e862b
Fast-forward
swh/provenance/__init__.py | 98 +++--
swh/provenance/archive.py | 25 +-
swh/provenance/backend.py | 322 ++++++++++++++++
swh/provenance/cli.py | 80 ++--
swh/provenance/graph.py | 34 +-
swh/provenance/model.py | 91 +++--
swh/provenance/origin.py | 33 +-
swh/provenance/postgresql/archive.py | 130 +++----
swh/provenance/postgresql/db_utils.py | 61 ----
swh/provenance/postgresql/provenancedb_base.py | 404 ++++++++++++---------
.../postgresql/provenancedb_with_path.py | 117 +++---
.../postgresql/provenancedb_without_path.py | 96 ++---
swh/provenance/provenance.py | 387 ++++++++++----------
swh/provenance/revision.py | 43 +--
swh/provenance/sql/30-schema.sql | 30 +-
swh/provenance/storage/archive.py | 39 +-
swh/provenance/tests/conftest.py | 70 ++--
.../tests/data/generate_storage_from_git.py | 3 +-
.../data/history_graphs_with-merges_visits-01.yaml | 55 +++
swh/provenance/tests/data/with-merges.msgpack | Bin 0 -> 7501 bytes
...repo_with_merges.yaml => with-merges_repo.yaml} | 0
...s-visits-01.yaml => with-merges_visits-01.yaml} | 0
swh/provenance/tests/test_archive_interface.py | 51 +++
swh/provenance/tests/test_cli.py | 16 +-
swh/provenance/tests/test_conftest.py | 9 +-
swh/provenance/tests/test_history_graph.py | 74 ++++
swh/provenance/tests/test_isochrone_graph.py | 17 +-
swh/provenance/tests/test_origin_iterator.py | 23 +-
swh/provenance/tests/test_provenance_db.py | 29 +-
swh/provenance/tests/test_provenance_heuristics.py | 120 +++---
swh/provenance/tests/test_revision_iterator.py | 6 +-
31 files changed, 1504 insertions(+), 959 deletions(-)
create mode 100644 swh/provenance/backend.py
delete mode 100644 swh/provenance/postgresql/db_utils.py
create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
create mode 100644 swh/provenance/tests/data/with-merges.msgpack
rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
create mode 100644 swh/provenance/tests/test_archive_interface.py
create mode 100644 swh/provenance/tests/test_history_graph.pyChanges applied before test
commit b9e862ba663eb8faf0651f37ca0df041cd544083
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Jul 2 01:05:24 2021 +0200
Improve type annotations for test
Exhaustively added type annotations to all test functions (there are some pending
TODO though) and applied the necessary fixes for the code to be compliant.
commit 3cd0dd9fdd8dfc103599af401a3ad95c8a459c68
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 22:55:51 2021 +0200
Improve type annotations throughout the module
All main module's files were updated with exhaustive type annotations,
making the necessary (minor) fixes for the code to be compliant with it.
Also, `__future__. annotations` was imported to avoid using strings when
possible.
commit fec57362bf581758fb52ee1b3e6b908e5ee52a51
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 17:48:12 2021 +0200
Improve typing annotations for `origin` and `revision` modules
This implie fixing `RevisionCSVIterator`, and updating cli and related tests.
commit a4eb765f92d25d45ddea178aaeebc375d69e3830
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 17:34:08 2021 +0200
Remove `db_utils` in favour of `swh.core.db.BaseDb` methods
This allows to remove some unnecessary `psycopg2` dependencies in modules that
don't deal directly with SQL queries.
commit 2a1aeb023627ed31ee4eea617790607e1fa1ba68
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Jun 29 14:28:54 2021 +0200
Force `snapshot_get_heads` to return revisions in chronological order
commit f819e4332df40b1ef35ff737f2558de570379473
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 19:21:58 2021 +0200
Add `ProvenanceStorageInterface` as discussed during backend design
Rework backend-related classes to properly use the new interface.
Adapt tests to the new structure as well.
commit 3672235c3258cf93fb37a82d060bf40ba1761b8b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 14:37:50 2021 +0200
Move `ProvenanceBackend` implementation to a separate file
commit 6f4da6fed7e663273627ad4a46c8489ef0a0e784
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 13:47:26 2021 +0200
Use `RealDictCursor` in `ProvenanceDBBase`
to improve the way `ProvenanceResult`s are generated.
commit 07a30e43a76e170ab03764035da68dcf7db1fc3b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 14:28:32 2021 +0200
Rework `ProvenanceInterface` as discussed during backend design
Add `ProvenanceResult` class to be returned by `content_find_first` and
`content_find_all` methods. Rename some methods. Improve type annotations.
commit 2fd3f56b57f8db6691ae6b8b7cb7ac557b764172
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:31:16 2021 +0200
Add tests for history graph topology
commit d45d6ff9e9317ecfe38d584df7297c548b654d28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:10:38 2021 +0200
Fix database queries related to the origin-revision layer
This required allowing null dates in the `revision` table so that revision can be added
by the origin-revision layer algorithm but not recognized as already processed by the
revision-content layer. Revision and origin entries are now inserted in the database
prior to inserting rows to revision_in_origin and revision_before_revision relations,
so that internal ids are properly resolved.
commit 0e2a3c64ce3c368b53c101c541e8aebcde789477
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Jun 25 13:38:26 2021 +0200
Add test to compare both `ArchiveInterface` implementations
Improve documentation of the interface and complete pending TODO's.
commit 98bba93cccece2b47ec4cd5887997cb5bede1e87
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:25:15 2021 +0200
Rename test files to keep naming convension
Also added missing .msgpack file dump for new with-merges repository.
commit fa9198afb71bcf3b8abea07d88d763a430f7358e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:05:24 2021 +0200
Refactor `ArchiveInterface` to fit origin-revision layer needs
Replace `revision_get` method by `revision_get_parents` returning an iterable of
parents' ids only, instead of a swh.model.model.Revision object.
commit 9e0c1aa099073887206c9334e17b49ee31bbef9a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Jun 23 20:00:40 2021 +0200
Use `Sha1Git` type to explicitly state the kind of identifiers
Previous occurrences of `bytes` and `Sha1` are now correctly using `Sha1Git`.
Also, some bytes conversion methods were replaced by their counterparts in
the swh.model.hashutil module.
commit a27ffff67b6b14bf37d153bb9b1d1c2ae63773fc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Jun 23 19:12:06 2021 +0200
Add support for sha1 identifiers for originsSee https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/238/ for more details.
| swh/provenance/tests/test_origin_iterator.py | ||
|---|---|---|
| 22–27 | Good point, I don't really know the reason behind this being an optional value but what I'm doing here is just mimic how we iterate OriginRevisionStatuses in our Archive classes. The point is that i don't want the test to fail is visit.visit is None, I just want this visit to be ignored. | |
| swh/provenance/tests/test_provenance_db.py | ||
| 35–37 | I don't know the impact of such change, but I agree it is a good idea | |
Comment Actions
Build is green
Patch application report for D5962 (id=21440)
Could not rebase; Attempt merge onto d892b29e40...
Updating d892b29..c57d8b9
Fast-forward
swh/provenance/__init__.py | 98 +++--
swh/provenance/archive.py | 25 +-
swh/provenance/backend.py | 322 ++++++++++++++++
swh/provenance/cli.py | 80 ++--
swh/provenance/graph.py | 34 +-
swh/provenance/model.py | 91 +++--
swh/provenance/origin.py | 33 +-
swh/provenance/postgresql/archive.py | 130 +++----
swh/provenance/postgresql/db_utils.py | 61 ----
swh/provenance/postgresql/provenancedb_base.py | 404 ++++++++++++---------
.../postgresql/provenancedb_with_path.py | 117 +++---
.../postgresql/provenancedb_without_path.py | 96 ++---
swh/provenance/provenance.py | 387 ++++++++++----------
swh/provenance/revision.py | 43 +--
swh/provenance/sql/30-schema.sql | 30 +-
swh/provenance/storage/archive.py | 39 +-
swh/provenance/tests/conftest.py | 70 ++--
.../tests/data/generate_storage_from_git.py | 3 +-
.../data/history_graphs_with-merges_visits-01.yaml | 55 +++
swh/provenance/tests/data/with-merges.msgpack | Bin 0 -> 7501 bytes
...repo_with_merges.yaml => with-merges_repo.yaml} | 0
...s-visits-01.yaml => with-merges_visits-01.yaml} | 0
swh/provenance/tests/test_archive_interface.py | 51 +++
swh/provenance/tests/test_cli.py | 16 +-
swh/provenance/tests/test_conftest.py | 9 +-
swh/provenance/tests/test_history_graph.py | 74 ++++
swh/provenance/tests/test_isochrone_graph.py | 17 +-
swh/provenance/tests/test_origin_iterator.py | 23 +-
swh/provenance/tests/test_provenance_db.py | 29 +-
swh/provenance/tests/test_provenance_heuristics.py | 120 +++---
swh/provenance/tests/test_revision_iterator.py | 6 +-
31 files changed, 1504 insertions(+), 959 deletions(-)
create mode 100644 swh/provenance/backend.py
delete mode 100644 swh/provenance/postgresql/db_utils.py
create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
create mode 100644 swh/provenance/tests/data/with-merges.msgpack
rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
create mode 100644 swh/provenance/tests/test_archive_interface.py
create mode 100644 swh/provenance/tests/test_history_graph.pyChanges applied before test
commit c57d8b960e912722379ee0ec4a4177267abc81eb
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Jul 2 01:05:24 2021 +0200
Improve type annotations for test
Exhaustively added type annotations to all test functions (there are some pending
TODO though) and applied the necessary fixes for the code to be compliant.
commit 54d6ba5d12a7c6f5181cb61b9c34f7e6f3b3a78b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 22:55:51 2021 +0200
Improve type annotations throughout the module
All main module's files were updated with exhaustive type annotations,
making the necessary (minor) fixes for the code to be compliant with it.
Also, `__future__. annotations` was imported to avoid using strings when
possible.
commit fec57362bf581758fb52ee1b3e6b908e5ee52a51
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 17:48:12 2021 +0200
Improve typing annotations for `origin` and `revision` modules
This implie fixing `RevisionCSVIterator`, and updating cli and related tests.
commit a4eb765f92d25d45ddea178aaeebc375d69e3830
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 17:34:08 2021 +0200
Remove `db_utils` in favour of `swh.core.db.BaseDb` methods
This allows to remove some unnecessary `psycopg2` dependencies in modules that
don't deal directly with SQL queries.
commit 2a1aeb023627ed31ee4eea617790607e1fa1ba68
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Jun 29 14:28:54 2021 +0200
Force `snapshot_get_heads` to return revisions in chronological order
commit f819e4332df40b1ef35ff737f2558de570379473
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 19:21:58 2021 +0200
Add `ProvenanceStorageInterface` as discussed during backend design
Rework backend-related classes to properly use the new interface.
Adapt tests to the new structure as well.
commit 3672235c3258cf93fb37a82d060bf40ba1761b8b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 14:37:50 2021 +0200
Move `ProvenanceBackend` implementation to a separate file
commit 6f4da6fed7e663273627ad4a46c8489ef0a0e784
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 13:47:26 2021 +0200
Use `RealDictCursor` in `ProvenanceDBBase`
to improve the way `ProvenanceResult`s are generated.
commit 07a30e43a76e170ab03764035da68dcf7db1fc3b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 14:28:32 2021 +0200
Rework `ProvenanceInterface` as discussed during backend design
Add `ProvenanceResult` class to be returned by `content_find_first` and
`content_find_all` methods. Rename some methods. Improve type annotations.
commit 2fd3f56b57f8db6691ae6b8b7cb7ac557b764172
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:31:16 2021 +0200
Add tests for history graph topology
commit d45d6ff9e9317ecfe38d584df7297c548b654d28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:10:38 2021 +0200
Fix database queries related to the origin-revision layer
This required allowing null dates in the `revision` table so that revision can be added
by the origin-revision layer algorithm but not recognized as already processed by the
revision-content layer. Revision and origin entries are now inserted in the database
prior to inserting rows to revision_in_origin and revision_before_revision relations,
so that internal ids are properly resolved.
commit 0e2a3c64ce3c368b53c101c541e8aebcde789477
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Jun 25 13:38:26 2021 +0200
Add test to compare both `ArchiveInterface` implementations
Improve documentation of the interface and complete pending TODO's.
commit 98bba93cccece2b47ec4cd5887997cb5bede1e87
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:25:15 2021 +0200
Rename test files to keep naming convension
Also added missing .msgpack file dump for new with-merges repository.
commit fa9198afb71bcf3b8abea07d88d763a430f7358e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:05:24 2021 +0200
Refactor `ArchiveInterface` to fit origin-revision layer needs
Replace `revision_get` method by `revision_get_parents` returning an iterable of
parents' ids only, instead of a swh.model.model.Revision object.
commit 9e0c1aa099073887206c9334e17b49ee31bbef9a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Jun 23 20:00:40 2021 +0200
Use `Sha1Git` type to explicitly state the kind of identifiers
Previous occurrences of `bytes` and `Sha1` are now correctly using `Sha1Git`.
Also, some bytes conversion methods were replaced by their counterparts in
the swh.model.hashutil module.
commit a27ffff67b6b14bf37d153bb9b1d1c2ae63773fc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Jun 23 19:12:06 2021 +0200
Add support for sha1 identifiers for originsSee https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/240/ for more details.
Comment Actions
Build is green
Patch application report for D5962 (id=21451)
Could not rebase; Attempt merge onto d892b29e40...
Updating d892b29..2e46f7c
Fast-forward
swh/provenance/__init__.py | 98 +++--
swh/provenance/archive.py | 25 +-
swh/provenance/backend.py | 324 +++++++++++++++++
swh/provenance/cli.py | 80 ++--
swh/provenance/graph.py | 34 +-
swh/provenance/model.py | 91 +++--
swh/provenance/origin.py | 33 +-
swh/provenance/postgresql/archive.py | 130 +++----
swh/provenance/postgresql/db_utils.py | 61 ----
swh/provenance/postgresql/provenancedb_base.py | 404 ++++++++++++---------
.../postgresql/provenancedb_with_path.py | 117 +++---
.../postgresql/provenancedb_without_path.py | 96 ++---
swh/provenance/provenance.py | 387 ++++++++++----------
swh/provenance/revision.py | 43 +--
swh/provenance/sql/30-schema.sql | 30 +-
swh/provenance/storage/archive.py | 39 +-
swh/provenance/tests/conftest.py | 70 ++--
.../tests/data/generate_storage_from_git.py | 3 +-
.../data/history_graphs_with-merges_visits-01.yaml | 55 +++
swh/provenance/tests/data/with-merges.msgpack | Bin 0 -> 7501 bytes
...repo_with_merges.yaml => with-merges_repo.yaml} | 0
...s-visits-01.yaml => with-merges_visits-01.yaml} | 0
swh/provenance/tests/test_archive_interface.py | 51 +++
swh/provenance/tests/test_cli.py | 16 +-
swh/provenance/tests/test_conftest.py | 9 +-
swh/provenance/tests/test_history_graph.py | 74 ++++
swh/provenance/tests/test_isochrone_graph.py | 17 +-
swh/provenance/tests/test_origin_iterator.py | 23 +-
swh/provenance/tests/test_provenance_db.py | 29 +-
swh/provenance/tests/test_provenance_heuristics.py | 120 +++---
swh/provenance/tests/test_revision_iterator.py | 6 +-
31 files changed, 1506 insertions(+), 959 deletions(-)
create mode 100644 swh/provenance/backend.py
delete mode 100644 swh/provenance/postgresql/db_utils.py
create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
create mode 100644 swh/provenance/tests/data/with-merges.msgpack
rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
create mode 100644 swh/provenance/tests/test_archive_interface.py
create mode 100644 swh/provenance/tests/test_history_graph.pyChanges applied before test
commit 2e46f7c34b0b5ddeb1a09b81e059029368c8226f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Jul 2 01:05:24 2021 +0200
Improve type annotations for test
Exhaustively added type annotations to all test functions (there are some pending
TODO though) and applied the necessary fixes for the code to be compliant.
commit 43238095b057ff9a0bb1ff001ae47d003fb9f2ff
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 22:55:51 2021 +0200
Improve type annotations throughout the module
All main module's files were updated with exhaustive type annotations,
making the necessary (minor) fixes for the code to be compliant with it.
Also, `__future__. annotations` was imported to avoid using strings when
possible.
commit 2483b17fb2a8328c997c068d68c65975f2d7f9fb
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 17:48:12 2021 +0200
Improve typing annotations for `origin` and `revision` modules
This implie fixing `RevisionCSVIterator`, and updating cli and related tests.
commit 6617ca5615ec000b90890cbf51979c0311ff1ea5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 17:34:08 2021 +0200
Remove `db_utils` in favour of `swh.core.db.BaseDb` methods
This allows to remove some unnecessary `psycopg2` dependencies in modules that
don't deal directly with SQL queries.
commit c72b2e2428fe410c17220648d378c505bba35350
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Jun 29 14:28:54 2021 +0200
Force `snapshot_get_heads` to return revisions in chronological order
commit 799839120cb99f22ce4272468ae0e388c335fb06
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 19:21:58 2021 +0200
Add `ProvenanceStorageInterface` as discussed during backend design
Rework backend-related classes to properly use the new interface.
Adapt tests to the new structure as well.
commit 7c0a091ce5ffbf0a02dbe9d7fc84435ddd46cde2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 14:37:50 2021 +0200
Move `ProvenanceBackend` implementation to a separate file
commit 34898ad3cb18c24a7d7bef79dcfe470c3a1374ef
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jul 1 13:47:26 2021 +0200
Use `RealDictCursor` in `ProvenanceDBBase`
to improve the way `ProvenanceResult`s are generated.
commit 721354c436b5f5a861800b11e6151afa1aa634b6
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Mon Jun 28 14:28:32 2021 +0200
Rework `ProvenanceInterface` as discussed during backend design
Add `ProvenanceResult` class to be returned by `content_find_first` and
`content_find_all` methods. Rename some methods. Improve type annotations.
commit 01f8d40ffccbcab6ecec6c2cf85478364e006caa
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:31:16 2021 +0200
Add tests for history graph topology
commit b7fdcdec7ea96101d62a57d9aeed114c897df961
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:10:38 2021 +0200
Fix database queries related to the origin-revision layer
This required allowing null dates in the `revision` table so that revision can be added
by the origin-revision layer algorithm but not recognized as already processed by the
revision-content layer. Revision and origin entries are now inserted in the database
prior to inserting rows to revision_in_origin and revision_before_revision relations,
so that internal ids are properly resolved.
commit 0e2a3c64ce3c368b53c101c541e8aebcde789477
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Jun 25 13:38:26 2021 +0200
Add test to compare both `ArchiveInterface` implementations
Improve documentation of the interface and complete pending TODO's.
commit 98bba93cccece2b47ec4cd5887997cb5bede1e87
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:25:15 2021 +0200
Rename test files to keep naming convension
Also added missing .msgpack file dump for new with-merges repository.
commit fa9198afb71bcf3b8abea07d88d763a430f7358e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Thu Jun 24 16:05:24 2021 +0200
Refactor `ArchiveInterface` to fit origin-revision layer needs
Replace `revision_get` method by `revision_get_parents` returning an iterable of
parents' ids only, instead of a swh.model.model.Revision object.
commit 9e0c1aa099073887206c9334e17b49ee31bbef9a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Jun 23 20:00:40 2021 +0200
Use `Sha1Git` type to explicitly state the kind of identifiers
Previous occurrences of `bytes` and `Sha1` are now correctly using `Sha1Git`.
Also, some bytes conversion methods were replaced by their counterparts in
the swh.model.hashutil module.
commit a27ffff67b6b14bf37d153bb9b1d1c2ae63773fc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Jun 23 19:12:06 2021 +0200
Add support for sha1 identifiers for originsSee https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/249/ for more details.