Details

Reviewers

douardda

Group Reviewers

Reviewers

Commits

rDPROVc72b2e2428fe: Force `snapshot_get_heads` to return revisions in chronological order

Summary

Depends on D5947

Diff Detail

Repository

rDPROV Provenance database

Branch

master

Lint

Lint Skipped

Unit

Unit Tests Skipped

Build Status

Buildable 22425
Build 34939: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 34938: arc lint + arc unit

Event Timeline

aeviso created this revision.Jun 29 2021, 2:30 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJun 29 2021, 2:30 PM

Build is green

Patch application report for D5948 (id=21349)

Could not rebase; Attempt merge onto d892b29e40...

Updating d892b29..2e61c99
Fast-forward
 swh/provenance/__init__.py                         |  47 ++-
 swh/provenance/archive.py                          |  25 +-
 swh/provenance/backend.py                          | 357 ++++++++++++++++++
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/model.py                            |  53 ++-
 swh/provenance/origin.py                           |  21 +-
 swh/provenance/postgresql/archive.py               | 117 +++---
 swh/provenance/postgresql/provenancedb_base.py     | 411 +++++++++++++--------
 .../postgresql/provenancedb_with_path.py           | 103 ++----
 .../postgresql/provenancedb_without_path.py        |  82 ++--
 swh/provenance/provenance.py                       | 257 ++++---------
 swh/provenance/revision.py                         |  13 +-
 swh/provenance/sql/30-schema.sql                   |  30 +-
 swh/provenance/storage/archive.py                  |  39 +-
 swh/provenance/tests/conftest.py                   |  32 +-
 .../tests/data/generate_storage_from_git.py        |   3 +-
 .../data/history_graphs_with-merges_visits-01.yaml |  55 +++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 0 -> 7501 bytes
 ...repo_with_merges.yaml => with-merges_repo.yaml} |   0
 ...s-visits-01.yaml => with-merges_visits-01.yaml} |   0
 swh/provenance/tests/test_archive_interface.py     |  51 +++
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |  62 ++++
 swh/provenance/tests/test_origin_iterator.py       |   8 +-
 swh/provenance/tests/test_provenance_db.py         |   4 +-
 swh/provenance/tests/test_provenance_heuristics.py |  30 +-
 26 files changed, 1126 insertions(+), 680 deletions(-)
 create mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
 create mode 100644 swh/provenance/tests/data/with-merges.msgpack
 rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
 rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
 create mode 100644 swh/provenance/tests/test_archive_interface.py
 create mode 100644 swh/provenance/tests/test_history_graph.py

Changes applied before test

commit 2e61c99fd23f264baa4843a323ecd973504492c8
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jun 29 14:28:54 2021 +0200

    Force `snapshot_get_heads` to return revisions in chronological order

commit 6afc8b39601c0f93375bcf37daa1b8a3d5bf242a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 19:21:58 2021 +0200

    Add ProvenanceStorageInterface
    
    Rework backend-related classes to properly use the new interface.
    Adapt tests to the new structure as well.

commit 23184e7de91d7e60577ce730868098b91a72b1d1
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:37:50 2021 +0200

    Move `ProvenanceBackend` implementation to a separate file

commit f32475952907452f3dbe3d51be9433aa854413bf
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:28:32 2021 +0200

    Rework `ProvenanceInterface` as discussed during backend design
    
    Add `ProvenanceResult` class to be returned by `content_find_first` and
    `content_find_all` methods. Rename some methods. Improve type annotations.

commit ad860db9bfeff7f276b3e356c9e21cb57cafc4c2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:31:16 2021 +0200

    Add tests for history graph topology

commit 37ac81faf15a32c4471a3c4ee5140bcb9bf57178
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:10:38 2021 +0200

    Fix database queries related to the origin-revision layer
    
    This required allowing null dates in the `revision` table so that revision can be added
    by the origin-revision layer algorithm but not recognized as already processed by the
    revision-content layer. Revision and origin entries are now inserted in the database
    prior to inserting rows to revision_in_origin and revision_before_revision relations,
    so that internal ids are properly resolved.

commit 4eb166cc4f2aa036c932b9a5eb462454a70ee0d9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jun 25 13:38:26 2021 +0200

    Add test to compare both ArchiveInterface implementations
    
    Improve documentation of the interface and complete pending TODO's.

commit 01ac9eea375258ac1e000389d3fd286d0dbae458
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:25:15 2021 +0200

    Rename test files to keep naming convension
    
    Also added missing .msgpack file dump for new with-merges repository.

commit 76d1560924251396c1ac63c286d8612ce0f7e9d9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:05:24 2021 +0200

    Refactor ArchiveInterface to fit origin-revision layer needs
    
    Replace `revision_get` method by `revision_get_parents` returning an iterable of
    parents' ids only, instead of a swh.model.model.Revision object.

commit df69a9e57692ed9d4d870c295a21b3ac187d7b9c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 20:00:40 2021 +0200

    Use `Sha1Git` type to explicitly state the kind of identifiers
    
    Previous occurrences of `bytes` and `Sha1` are not correctly using `Sha1Git`.
    Also, some bytes conversion methods were replaced by their counterparts in
    the swh.model.hashutil module.

commit fa22dc902781e30e46823030681f003983cc6d6e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 19:12:06 2021 +0200

    Add support for sha1 identifiers for origins

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/197/ for more details.

Harbormaster completed remote builds in B22328: Diff 21349.Jun 29 2021, 2:35 PM

aeviso requested review of this revision.Jun 29 2021, 2:35 PM

rebase

Build is green

Patch application report for D5948 (id=21354)

Could not rebase; Attempt merge onto d892b29e40...

Updating d892b29..da03485
Fast-forward
 swh/provenance/__init__.py                         |  47 ++-
 swh/provenance/archive.py                          |  25 +-
 swh/provenance/backend.py                          | 357 ++++++++++++++++++
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/model.py                            |  53 ++-
 swh/provenance/origin.py                           |  21 +-
 swh/provenance/postgresql/archive.py               | 117 +++---
 swh/provenance/postgresql/provenancedb_base.py     | 411 +++++++++++++--------
 .../postgresql/provenancedb_with_path.py           | 103 ++----
 .../postgresql/provenancedb_without_path.py        |  82 ++--
 swh/provenance/provenance.py                       | 298 ++++++---------
 swh/provenance/revision.py                         |  13 +-
 swh/provenance/sql/30-schema.sql                   |  30 +-
 swh/provenance/storage/archive.py                  |  39 +-
 swh/provenance/tests/conftest.py                   |  32 +-
 .../tests/data/generate_storage_from_git.py        |   3 +-
 .../data/history_graphs_with-merges_visits-01.yaml |  55 +++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 0 -> 7501 bytes
 ...repo_with_merges.yaml => with-merges_repo.yaml} |   0
 ...s-visits-01.yaml => with-merges_visits-01.yaml} |   0
 swh/provenance/tests/test_archive_interface.py     |  51 +++
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |  62 ++++
 swh/provenance/tests/test_origin_iterator.py       |   8 +-
 swh/provenance/tests/test_provenance_db.py         |   4 +-
 swh/provenance/tests/test_provenance_heuristics.py |  30 +-
 26 files changed, 1167 insertions(+), 680 deletions(-)
 create mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
 create mode 100644 swh/provenance/tests/data/with-merges.msgpack
 rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
 rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
 create mode 100644 swh/provenance/tests/test_archive_interface.py
 create mode 100644 swh/provenance/tests/test_history_graph.py

Changes applied before test

commit da03485a327a3fdad17bf85d383742a786f3f112
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jun 29 14:28:54 2021 +0200

    Force `snapshot_get_heads` to return revisions in chronological order

commit a3da0612eae1ded260eeafee9dc77f2bbf84a47f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 19:21:58 2021 +0200

    Add `ProvenanceStorageInterface` as discussed during backend design
    
    Rework backend-related classes to properly use the new interface.
    Adapt tests to the new structure as well.

commit 361199109d7d5a6cb694685cb2062940abe814bb
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:37:50 2021 +0200

    Move `ProvenanceBackend` implementation to a separate file

commit d058de2c080ee0c79ae57131d5c8ebdbeb6d0486
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:28:32 2021 +0200

    Rework `ProvenanceInterface` as discussed during backend design
    
    Add `ProvenanceResult` class to be returned by `content_find_first` and
    `content_find_all` methods. Rename some methods. Improve type annotations.

commit ad860db9bfeff7f276b3e356c9e21cb57cafc4c2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:31:16 2021 +0200

    Add tests for history graph topology

commit 37ac81faf15a32c4471a3c4ee5140bcb9bf57178
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:10:38 2021 +0200

    Fix database queries related to the origin-revision layer
    
    This required allowing null dates in the `revision` table so that revision can be added
    by the origin-revision layer algorithm but not recognized as already processed by the
    revision-content layer. Revision and origin entries are now inserted in the database
    prior to inserting rows to revision_in_origin and revision_before_revision relations,
    so that internal ids are properly resolved.

commit 4eb166cc4f2aa036c932b9a5eb462454a70ee0d9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jun 25 13:38:26 2021 +0200

    Add test to compare both ArchiveInterface implementations
    
    Improve documentation of the interface and complete pending TODO's.

commit 01ac9eea375258ac1e000389d3fd286d0dbae458
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:25:15 2021 +0200

    Rename test files to keep naming convension
    
    Also added missing .msgpack file dump for new with-merges repository.

commit 76d1560924251396c1ac63c286d8612ce0f7e9d9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:05:24 2021 +0200

    Refactor ArchiveInterface to fit origin-revision layer needs
    
    Replace `revision_get` method by `revision_get_parents` returning an iterable of
    parents' ids only, instead of a swh.model.model.Revision object.

commit df69a9e57692ed9d4d870c295a21b3ac187d7b9c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 20:00:40 2021 +0200

    Use `Sha1Git` type to explicitly state the kind of identifiers
    
    Previous occurrences of `bytes` and `Sha1` are not correctly using `Sha1Git`.
    Also, some bytes conversion methods were replaced by their counterparts in
    the swh.model.hashutil module.

commit fa22dc902781e30e46823030681f003983cc6d6e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 19:12:06 2021 +0200

    Add support for sha1 identifiers for origins

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/200/ for more details.

Harbormaster completed remote builds in B22334: Diff 21354.Jun 29 2021, 4:00 PM

rebase

Build is green

Patch application report for D5948 (id=21365)

Could not rebase; Attempt merge onto d892b29e40...

Updating d892b29..c38537f
Fast-forward
 swh/provenance/__init__.py                         |  47 ++-
 swh/provenance/archive.py                          |  25 +-
 swh/provenance/backend.py                          | 357 ++++++++++++++++++
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/model.py                            |  53 ++-
 swh/provenance/origin.py                           |  21 +-
 swh/provenance/postgresql/archive.py               | 117 +++---
 swh/provenance/postgresql/provenancedb_base.py     | 398 ++++++++++++---------
 .../postgresql/provenancedb_with_path.py           | 115 +++---
 .../postgresql/provenancedb_without_path.py        |  94 ++---
 swh/provenance/provenance.py                       | 341 ++++++++----------
 swh/provenance/revision.py                         |  13 +-
 swh/provenance/sql/30-schema.sql                   |  30 +-
 swh/provenance/storage/archive.py                  |  39 +-
 swh/provenance/tests/conftest.py                   |  32 +-
 .../tests/data/generate_storage_from_git.py        |   3 +-
 .../data/history_graphs_with-merges_visits-01.yaml |  55 +++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 0 -> 7501 bytes
 ...repo_with_merges.yaml => with-merges_repo.yaml} |   0
 ...s-visits-01.yaml => with-merges_visits-01.yaml} |   0
 swh/provenance/tests/test_archive_interface.py     |  51 +++
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |  62 ++++
 swh/provenance/tests/test_origin_iterator.py       |   8 +-
 swh/provenance/tests/test_provenance_db.py         |   4 +-
 swh/provenance/tests/test_provenance_heuristics.py |  30 +-
 26 files changed, 1199 insertions(+), 702 deletions(-)
 create mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
 create mode 100644 swh/provenance/tests/data/with-merges.msgpack
 rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
 rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
 create mode 100644 swh/provenance/tests/test_archive_interface.py
 create mode 100644 swh/provenance/tests/test_history_graph.py

Changes applied before test

commit c38537f2f97808864251898ae7aa9613bad530b4
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jun 29 14:28:54 2021 +0200

    Force `snapshot_get_heads` to return revisions in chronological order

commit 695b498600682045004fdd04859ecf9e96819479
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 19:21:58 2021 +0200

    Add `ProvenanceStorageInterface` as discussed during backend design
    
    Rework backend-related classes to properly use the new interface.
    Adapt tests to the new structure as well.

commit 361199109d7d5a6cb694685cb2062940abe814bb
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:37:50 2021 +0200

    Move `ProvenanceBackend` implementation to a separate file

commit d058de2c080ee0c79ae57131d5c8ebdbeb6d0486
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:28:32 2021 +0200

    Rework `ProvenanceInterface` as discussed during backend design
    
    Add `ProvenanceResult` class to be returned by `content_find_first` and
    `content_find_all` methods. Rename some methods. Improve type annotations.

commit ad860db9bfeff7f276b3e356c9e21cb57cafc4c2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:31:16 2021 +0200

    Add tests for history graph topology

commit 37ac81faf15a32c4471a3c4ee5140bcb9bf57178
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:10:38 2021 +0200

    Fix database queries related to the origin-revision layer
    
    This required allowing null dates in the `revision` table so that revision can be added
    by the origin-revision layer algorithm but not recognized as already processed by the
    revision-content layer. Revision and origin entries are now inserted in the database
    prior to inserting rows to revision_in_origin and revision_before_revision relations,
    so that internal ids are properly resolved.

commit 4eb166cc4f2aa036c932b9a5eb462454a70ee0d9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jun 25 13:38:26 2021 +0200

    Add test to compare both ArchiveInterface implementations
    
    Improve documentation of the interface and complete pending TODO's.

commit 01ac9eea375258ac1e000389d3fd286d0dbae458
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:25:15 2021 +0200

    Rename test files to keep naming convension
    
    Also added missing .msgpack file dump for new with-merges repository.

commit 76d1560924251396c1ac63c286d8612ce0f7e9d9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:05:24 2021 +0200

    Refactor ArchiveInterface to fit origin-revision layer needs
    
    Replace `revision_get` method by `revision_get_parents` returning an iterable of
    parents' ids only, instead of a swh.model.model.Revision object.

commit df69a9e57692ed9d4d870c295a21b3ac187d7b9c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 20:00:40 2021 +0200

    Use `Sha1Git` type to explicitly state the kind of identifiers
    
    Previous occurrences of `bytes` and `Sha1` are not correctly using `Sha1Git`.
    Also, some bytes conversion methods were replaced by their counterparts in
    the swh.model.hashutil module.

commit fa22dc902781e30e46823030681f003983cc6d6e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 19:12:06 2021 +0200

    Add support for sha1 identifiers for origins

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/202/ for more details.

Harbormaster completed remote builds in B22344: Diff 21365.Jun 30 2021, 10:19 AM

rebase

Build is green

Patch application report for D5948 (id=21371)

Could not rebase; Attempt merge onto d892b29e40...

Updating d892b29..8d53887
Fast-forward
 swh/provenance/__init__.py                         |  47 ++-
 swh/provenance/archive.py                          |  25 +-
 swh/provenance/backend.py                          | 357 ++++++++++++++++++
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/model.py                            |  53 ++-
 swh/provenance/origin.py                           |  21 +-
 swh/provenance/postgresql/archive.py               | 117 +++---
 swh/provenance/postgresql/provenancedb_base.py     | 398 ++++++++++++---------
 .../postgresql/provenancedb_with_path.py           | 115 +++---
 .../postgresql/provenancedb_without_path.py        |  94 ++---
 swh/provenance/provenance.py                       | 341 ++++++++----------
 swh/provenance/revision.py                         |  13 +-
 swh/provenance/sql/30-schema.sql                   |  30 +-
 swh/provenance/storage/archive.py                  |  39 +-
 swh/provenance/tests/conftest.py                   |  32 +-
 .../tests/data/generate_storage_from_git.py        |   3 +-
 .../data/history_graphs_with-merges_visits-01.yaml |  55 +++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 0 -> 7501 bytes
 ...repo_with_merges.yaml => with-merges_repo.yaml} |   0
 ...s-visits-01.yaml => with-merges_visits-01.yaml} |   0
 swh/provenance/tests/test_archive_interface.py     |  51 +++
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |  62 ++++
 swh/provenance/tests/test_origin_iterator.py       |   8 +-
 swh/provenance/tests/test_provenance_db.py         |   4 +-
 swh/provenance/tests/test_provenance_heuristics.py |  30 +-
 26 files changed, 1199 insertions(+), 702 deletions(-)
 create mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
 create mode 100644 swh/provenance/tests/data/with-merges.msgpack
 rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
 rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
 create mode 100644 swh/provenance/tests/test_archive_interface.py
 create mode 100644 swh/provenance/tests/test_history_graph.py

Changes applied before test

commit 8d5388799022e6ea6f51e124cdf64c753af1397a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jun 29 14:28:54 2021 +0200

    Force `snapshot_get_heads` to return revisions in chronological order

commit 2304647745e79308b72978ba3a9141f3e6f844f8
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 19:21:58 2021 +0200

    Add `ProvenanceStorageInterface` as discussed during backend design
    
    Rework backend-related classes to properly use the new interface.
    Adapt tests to the new structure as well.

commit e25122d2e47de942a772164e9f1a60f425c87d97
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:37:50 2021 +0200

    Move `ProvenanceBackend` implementation to a separate file

commit b7678a341da72587cc48848f5a72f65861f892af
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:28:32 2021 +0200

    Rework `ProvenanceInterface` as discussed during backend design
    
    Add `ProvenanceResult` class to be returned by `content_find_first` and
    `content_find_all` methods. Rename some methods. Improve type annotations.

commit 7a59ff712bb8b5ae22e6f016475d03317c27b64a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:31:16 2021 +0200

    Add tests for history graph topology

commit 3171ae2f129df433689fd22e32c8eeebf7af4171
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:10:38 2021 +0200

    Fix database queries related to the origin-revision layer
    
    This required allowing null dates in the `revision` table so that revision can be added
    by the origin-revision layer algorithm but not recognized as already processed by the
    revision-content layer. Revision and origin entries are now inserted in the database
    prior to inserting rows to revision_in_origin and revision_before_revision relations,
    so that internal ids are properly resolved.

commit 6736f6068280f167df5616681dee9ad67b2b7dbd
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jun 25 13:38:26 2021 +0200

    Add test to compare both `ArchiveInterface` implementations
    
    Improve documentation of the interface and complete pending TODO's.

commit dde867254e51dd87f4aba3cdea59da8bffc2d160
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:25:15 2021 +0200

    Rename test files to keep naming convension
    
    Also added missing .msgpack file dump for new with-merges repository.

commit 14001c1844598a3d4ebd1b5f609070f9c85dcaa9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:05:24 2021 +0200

    Refactor `ArchiveInterface` to fit origin-revision layer needs
    
    Replace `revision_get` method by `revision_get_parents` returning an iterable of
    parents' ids only, instead of a swh.model.model.Revision object.

commit df69a9e57692ed9d4d870c295a21b3ac187d7b9c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 20:00:40 2021 +0200

    Use `Sha1Git` type to explicitly state the kind of identifiers
    
    Previous occurrences of `bytes` and `Sha1` are not correctly using `Sha1Git`.
    Also, some bytes conversion methods were replaced by their counterparts in
    the swh.model.hashutil module.

commit fa22dc902781e30e46823030681f003983cc6d6e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 19:12:06 2021 +0200

    Add support for sha1 identifiers for origins

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/208/ for more details.

Harbormaster completed remote builds in B22350: Diff 21371.Jun 30 2021, 10:43 AM

ok but the SQL query could be improved to not return unwanted dates

swh/provenance/postgresql/archive.py
110	(a bit nitpick but) since you do not need the dates to be returned, the SQL query should not select them.

This revision is now accepted and ready to land.Jul 1 2021, 10:49 AM

rebase

aeviso marked an inline comment as done.Jul 1 2021, 12:39 PM

Build is green

Patch application report for D5948 (id=21387)

Could not rebase; Attempt merge onto d892b29e40...

Updating d892b29..a569437
Fast-forward
 swh/provenance/__init__.py                         |  47 ++-
 swh/provenance/archive.py                          |  25 +-
 swh/provenance/backend.py                          | 357 ++++++++++++++++++
 swh/provenance/cli.py                              |  28 +-
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/model.py                            |  53 ++-
 swh/provenance/origin.py                           |  21 +-
 swh/provenance/postgresql/archive.py               | 117 +++---
 swh/provenance/postgresql/provenancedb_base.py     | 402 ++++++++++++---------
 .../postgresql/provenancedb_with_path.py           | 117 +++---
 .../postgresql/provenancedb_without_path.py        |  96 ++---
 swh/provenance/provenance.py                       | 349 +++++++++---------
 swh/provenance/revision.py                         |  13 +-
 swh/provenance/sql/30-schema.sql                   |  30 +-
 swh/provenance/storage/archive.py                  |  39 +-
 swh/provenance/tests/conftest.py                   |  32 +-
 .../tests/data/generate_storage_from_git.py        |   3 +-
 .../data/history_graphs_with-merges_visits-01.yaml |  55 +++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 0 -> 7501 bytes
 ...repo_with_merges.yaml => with-merges_repo.yaml} |   0
 ...s-visits-01.yaml => with-merges_visits-01.yaml} |   0
 swh/provenance/tests/test_archive_interface.py     |  51 +++
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |  62 ++++
 swh/provenance/tests/test_origin_iterator.py       |   8 +-
 swh/provenance/tests/test_provenance_db.py         |   4 +-
 swh/provenance/tests/test_provenance_heuristics.py |  56 +--
 27 files changed, 1229 insertions(+), 742 deletions(-)
 create mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
 create mode 100644 swh/provenance/tests/data/with-merges.msgpack
 rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
 rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
 create mode 100644 swh/provenance/tests/test_archive_interface.py
 create mode 100644 swh/provenance/tests/test_history_graph.py

Changes applied before test

commit a5694377e5e123eb9e79909380c113c07fa92b54
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jun 29 14:28:54 2021 +0200

    Force `snapshot_get_heads` to return revisions in chronological order

commit 35dafe5f8b1b95a0610199c93864ad16a1659283
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 1 12:07:41 2021 +0200

    Use `RealDictCursor` in `ProvenanceDBBase`
    
    to improve the way `ProvenanceResult`s are generated.
    
    Change `ProvenanceDBBase` from a `TypedDict` to a regular class.

commit 2304647745e79308b72978ba3a9141f3e6f844f8
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 19:21:58 2021 +0200

    Add `ProvenanceStorageInterface` as discussed during backend design
    
    Rework backend-related classes to properly use the new interface.
    Adapt tests to the new structure as well.

commit e25122d2e47de942a772164e9f1a60f425c87d97
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:37:50 2021 +0200

    Move `ProvenanceBackend` implementation to a separate file

commit b7678a341da72587cc48848f5a72f65861f892af
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:28:32 2021 +0200

    Rework `ProvenanceInterface` as discussed during backend design
    
    Add `ProvenanceResult` class to be returned by `content_find_first` and
    `content_find_all` methods. Rename some methods. Improve type annotations.

commit 7a59ff712bb8b5ae22e6f016475d03317c27b64a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:31:16 2021 +0200

    Add tests for history graph topology

commit 3171ae2f129df433689fd22e32c8eeebf7af4171
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:10:38 2021 +0200

    Fix database queries related to the origin-revision layer
    
    This required allowing null dates in the `revision` table so that revision can be added
    by the origin-revision layer algorithm but not recognized as already processed by the
    revision-content layer. Revision and origin entries are now inserted in the database
    prior to inserting rows to revision_in_origin and revision_before_revision relations,
    so that internal ids are properly resolved.

commit 6736f6068280f167df5616681dee9ad67b2b7dbd
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jun 25 13:38:26 2021 +0200

    Add test to compare both `ArchiveInterface` implementations
    
    Improve documentation of the interface and complete pending TODO's.

commit dde867254e51dd87f4aba3cdea59da8bffc2d160
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:25:15 2021 +0200

    Rename test files to keep naming convension
    
    Also added missing .msgpack file dump for new with-merges repository.

commit 14001c1844598a3d4ebd1b5f609070f9c85dcaa9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:05:24 2021 +0200

    Refactor `ArchiveInterface` to fit origin-revision layer needs
    
    Replace `revision_get` method by `revision_get_parents` returning an iterable of
    parents' ids only, instead of a swh.model.model.Revision object.

commit df69a9e57692ed9d4d870c295a21b3ac187d7b9c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 20:00:40 2021 +0200

    Use `Sha1Git` type to explicitly state the kind of identifiers
    
    Previous occurrences of `bytes` and `Sha1` are not correctly using `Sha1Git`.
    Also, some bytes conversion methods were replaced by their counterparts in
    the swh.model.hashutil module.

commit fa22dc902781e30e46823030681f003983cc6d6e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 19:12:06 2021 +0200

    Add support for sha1 identifiers for origins

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/210/ for more details.

Harbormaster completed remote builds in B22366: Diff 21387.Jul 1 2021, 12:40 PM

rebase

Build is green

Patch application report for D5948 (id=21388)

Could not rebase; Attempt merge onto d892b29e40...

Updating d892b29..6b0e59b
Fast-forward
 swh/provenance/__init__.py                         |  47 ++-
 swh/provenance/archive.py                          |  25 +-
 swh/provenance/backend.py                          | 357 ++++++++++++++++++
 swh/provenance/cli.py                              |  28 +-
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/model.py                            |  53 ++-
 swh/provenance/origin.py                           |  21 +-
 swh/provenance/postgresql/archive.py               | 127 +++----
 swh/provenance/postgresql/provenancedb_base.py     | 402 ++++++++++++---------
 .../postgresql/provenancedb_with_path.py           | 117 +++---
 .../postgresql/provenancedb_without_path.py        |  96 ++---
 swh/provenance/provenance.py                       | 349 +++++++++---------
 swh/provenance/revision.py                         |  13 +-
 swh/provenance/sql/30-schema.sql                   |  30 +-
 swh/provenance/storage/archive.py                  |  39 +-
 swh/provenance/tests/conftest.py                   |  32 +-
 .../tests/data/generate_storage_from_git.py        |   3 +-
 .../data/history_graphs_with-merges_visits-01.yaml |  55 +++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 0 -> 7501 bytes
 ...repo_with_merges.yaml => with-merges_repo.yaml} |   0
 ...s-visits-01.yaml => with-merges_visits-01.yaml} |   0
 swh/provenance/tests/test_archive_interface.py     |  51 +++
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |  62 ++++
 swh/provenance/tests/test_origin_iterator.py       |   8 +-
 swh/provenance/tests/test_provenance_db.py         |   4 +-
 swh/provenance/tests/test_provenance_heuristics.py |  56 +--
 27 files changed, 1239 insertions(+), 742 deletions(-)
 create mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
 create mode 100644 swh/provenance/tests/data/with-merges.msgpack
 rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
 rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
 create mode 100644 swh/provenance/tests/test_archive_interface.py
 create mode 100644 swh/provenance/tests/test_history_graph.py

Changes applied before test

commit 6b0e59bd18ac723020c34e537f8f250bbb75475e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jun 29 14:28:54 2021 +0200

    Force `snapshot_get_heads` to return revisions in chronological order

commit 35dafe5f8b1b95a0610199c93864ad16a1659283
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 1 12:07:41 2021 +0200

    Use `RealDictCursor` in `ProvenanceDBBase`
    
    to improve the way `ProvenanceResult`s are generated.
    
    Change `ProvenanceDBBase` from a `TypedDict` to a regular class.

commit 2304647745e79308b72978ba3a9141f3e6f844f8
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 19:21:58 2021 +0200

    Add `ProvenanceStorageInterface` as discussed during backend design
    
    Rework backend-related classes to properly use the new interface.
    Adapt tests to the new structure as well.

commit e25122d2e47de942a772164e9f1a60f425c87d97
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:37:50 2021 +0200

    Move `ProvenanceBackend` implementation to a separate file

commit b7678a341da72587cc48848f5a72f65861f892af
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:28:32 2021 +0200

    Rework `ProvenanceInterface` as discussed during backend design
    
    Add `ProvenanceResult` class to be returned by `content_find_first` and
    `content_find_all` methods. Rename some methods. Improve type annotations.

commit 7a59ff712bb8b5ae22e6f016475d03317c27b64a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:31:16 2021 +0200

    Add tests for history graph topology

commit 3171ae2f129df433689fd22e32c8eeebf7af4171
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:10:38 2021 +0200

    Fix database queries related to the origin-revision layer
    
    This required allowing null dates in the `revision` table so that revision can be added
    by the origin-revision layer algorithm but not recognized as already processed by the
    revision-content layer. Revision and origin entries are now inserted in the database
    prior to inserting rows to revision_in_origin and revision_before_revision relations,
    so that internal ids are properly resolved.

commit 6736f6068280f167df5616681dee9ad67b2b7dbd
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jun 25 13:38:26 2021 +0200

    Add test to compare both `ArchiveInterface` implementations
    
    Improve documentation of the interface and complete pending TODO's.

commit dde867254e51dd87f4aba3cdea59da8bffc2d160
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:25:15 2021 +0200

    Rename test files to keep naming convension
    
    Also added missing .msgpack file dump for new with-merges repository.

commit 14001c1844598a3d4ebd1b5f609070f9c85dcaa9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:05:24 2021 +0200

    Refactor `ArchiveInterface` to fit origin-revision layer needs
    
    Replace `revision_get` method by `revision_get_parents` returning an iterable of
    parents' ids only, instead of a swh.model.model.Revision object.

commit df69a9e57692ed9d4d870c295a21b3ac187d7b9c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 20:00:40 2021 +0200

    Use `Sha1Git` type to explicitly state the kind of identifiers
    
    Previous occurrences of `bytes` and `Sha1` are not correctly using `Sha1Git`.
    Also, some bytes conversion methods were replaced by their counterparts in
    the swh.model.hashutil module.

commit fa22dc902781e30e46823030681f003983cc6d6e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 19:12:06 2021 +0200

    Add support for sha1 identifiers for origins

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/211/ for more details.

Harbormaster completed remote builds in B22367: Diff 21388.Jul 1 2021, 1:23 PM

rebase

Build is green

Patch application report for D5948 (id=21389)

Could not rebase; Attempt merge onto d892b29e40...

Updating d892b29..e9b91a4
Fast-forward
 swh/provenance/__init__.py                         |  47 ++-
 swh/provenance/archive.py                          |  25 +-
 swh/provenance/backend.py                          | 357 ++++++++++++++++++
 swh/provenance/cli.py                              |  28 +-
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/model.py                            |  53 ++-
 swh/provenance/origin.py                           |  21 +-
 swh/provenance/postgresql/archive.py               | 126 +++----
 swh/provenance/postgresql/provenancedb_base.py     | 402 ++++++++++++---------
 .../postgresql/provenancedb_with_path.py           | 117 +++---
 .../postgresql/provenancedb_without_path.py        |  96 ++---
 swh/provenance/provenance.py                       | 349 +++++++++---------
 swh/provenance/revision.py                         |  13 +-
 swh/provenance/sql/30-schema.sql                   |  30 +-
 swh/provenance/storage/archive.py                  |  39 +-
 swh/provenance/tests/conftest.py                   |  32 +-
 .../tests/data/generate_storage_from_git.py        |   3 +-
 .../data/history_graphs_with-merges_visits-01.yaml |  55 +++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 0 -> 7501 bytes
 ...repo_with_merges.yaml => with-merges_repo.yaml} |   0
 ...s-visits-01.yaml => with-merges_visits-01.yaml} |   0
 swh/provenance/tests/test_archive_interface.py     |  51 +++
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |  62 ++++
 swh/provenance/tests/test_origin_iterator.py       |   8 +-
 swh/provenance/tests/test_provenance_db.py         |   4 +-
 swh/provenance/tests/test_provenance_heuristics.py |  56 +--
 27 files changed, 1238 insertions(+), 742 deletions(-)
 create mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
 create mode 100644 swh/provenance/tests/data/with-merges.msgpack
 rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
 rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
 create mode 100644 swh/provenance/tests/test_archive_interface.py
 create mode 100644 swh/provenance/tests/test_history_graph.py

Changes applied before test

commit e9b91a40802043fcf15f41f82faac80df6083066
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jun 29 14:28:54 2021 +0200

    Force `snapshot_get_heads` to return revisions in chronological order

commit 35dafe5f8b1b95a0610199c93864ad16a1659283
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 1 12:07:41 2021 +0200

    Use `RealDictCursor` in `ProvenanceDBBase`
    
    to improve the way `ProvenanceResult`s are generated.
    
    Change `ProvenanceDBBase` from a `TypedDict` to a regular class.

commit 2304647745e79308b72978ba3a9141f3e6f844f8
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 19:21:58 2021 +0200

    Add `ProvenanceStorageInterface` as discussed during backend design
    
    Rework backend-related classes to properly use the new interface.
    Adapt tests to the new structure as well.

commit e25122d2e47de942a772164e9f1a60f425c87d97
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:37:50 2021 +0200

    Move `ProvenanceBackend` implementation to a separate file

commit b7678a341da72587cc48848f5a72f65861f892af
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:28:32 2021 +0200

    Rework `ProvenanceInterface` as discussed during backend design
    
    Add `ProvenanceResult` class to be returned by `content_find_first` and
    `content_find_all` methods. Rename some methods. Improve type annotations.

commit 7a59ff712bb8b5ae22e6f016475d03317c27b64a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:31:16 2021 +0200

    Add tests for history graph topology

commit 3171ae2f129df433689fd22e32c8eeebf7af4171
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:10:38 2021 +0200

    Fix database queries related to the origin-revision layer
    
    This required allowing null dates in the `revision` table so that revision can be added
    by the origin-revision layer algorithm but not recognized as already processed by the
    revision-content layer. Revision and origin entries are now inserted in the database
    prior to inserting rows to revision_in_origin and revision_before_revision relations,
    so that internal ids are properly resolved.

commit 6736f6068280f167df5616681dee9ad67b2b7dbd
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jun 25 13:38:26 2021 +0200

    Add test to compare both `ArchiveInterface` implementations
    
    Improve documentation of the interface and complete pending TODO's.

commit dde867254e51dd87f4aba3cdea59da8bffc2d160
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:25:15 2021 +0200

    Rename test files to keep naming convension
    
    Also added missing .msgpack file dump for new with-merges repository.

commit 14001c1844598a3d4ebd1b5f609070f9c85dcaa9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:05:24 2021 +0200

    Refactor `ArchiveInterface` to fit origin-revision layer needs
    
    Replace `revision_get` method by `revision_get_parents` returning an iterable of
    parents' ids only, instead of a swh.model.model.Revision object.

commit df69a9e57692ed9d4d870c295a21b3ac187d7b9c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 20:00:40 2021 +0200

    Use `Sha1Git` type to explicitly state the kind of identifiers
    
    Previous occurrences of `bytes` and `Sha1` are not correctly using `Sha1Git`.
    Also, some bytes conversion methods were replaced by their counterparts in
    the swh.model.hashutil module.

commit fa22dc902781e30e46823030681f003983cc6d6e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 19:12:06 2021 +0200

    Add support for sha1 identifiers for origins

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/212/ for more details.

Harbormaster completed remote builds in B22368: Diff 21389.Jul 1 2021, 1:28 PM

rebase

Build is green

Patch application report for D5948 (id=21398)

Could not rebase; Attempt merge onto d892b29e40...

Updating d892b29..0c3c781
Fast-forward
 swh/provenance/__init__.py                         |  47 ++-
 swh/provenance/archive.py                          |  25 +-
 swh/provenance/backend.py                          | 357 ++++++++++++++++++
 swh/provenance/cli.py                              |  28 +-
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/model.py                            |  53 ++-
 swh/provenance/origin.py                           |  21 +-
 swh/provenance/postgresql/archive.py               | 126 +++----
 swh/provenance/postgresql/provenancedb_base.py     | 402 ++++++++++++---------
 .../postgresql/provenancedb_with_path.py           | 117 +++---
 .../postgresql/provenancedb_without_path.py        |  96 ++---
 swh/provenance/provenance.py                       | 349 +++++++++---------
 swh/provenance/revision.py                         |  13 +-
 swh/provenance/sql/30-schema.sql                   |  30 +-
 swh/provenance/storage/archive.py                  |  39 +-
 swh/provenance/tests/conftest.py                   |  32 +-
 .../tests/data/generate_storage_from_git.py        |   3 +-
 .../data/history_graphs_with-merges_visits-01.yaml |  55 +++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 0 -> 7501 bytes
 ...repo_with_merges.yaml => with-merges_repo.yaml} |   0
 ...s-visits-01.yaml => with-merges_visits-01.yaml} |   0
 swh/provenance/tests/test_archive_interface.py     |  51 +++
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |  62 ++++
 swh/provenance/tests/test_origin_iterator.py       |   8 +-
 swh/provenance/tests/test_provenance_db.py         |   4 +-
 swh/provenance/tests/test_provenance_heuristics.py |  56 +--
 27 files changed, 1236 insertions(+), 744 deletions(-)
 create mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
 create mode 100644 swh/provenance/tests/data/with-merges.msgpack
 rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
 rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
 create mode 100644 swh/provenance/tests/test_archive_interface.py
 create mode 100644 swh/provenance/tests/test_history_graph.py

Changes applied before test

commit 0c3c7817cfd267bf3b77ce2e594ea7feb3ec96d5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jun 29 14:28:54 2021 +0200

    Force `snapshot_get_heads` to return revisions in chronological order

commit afb67f665ab00c03c0ca33e96b1bfc109c827c58
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 19:21:58 2021 +0200

    Add `ProvenanceStorageInterface` as discussed during backend design
    
    Rework backend-related classes to properly use the new interface.
    Adapt tests to the new structure as well.

commit 3672235c3258cf93fb37a82d060bf40ba1761b8b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:37:50 2021 +0200

    Move `ProvenanceBackend` implementation to a separate file

commit 6f4da6fed7e663273627ad4a46c8489ef0a0e784
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 1 13:47:26 2021 +0200

    Use `RealDictCursor` in `ProvenanceDBBase`
    
    to improve the way `ProvenanceResult`s are generated.

commit 07a30e43a76e170ab03764035da68dcf7db1fc3b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:28:32 2021 +0200

    Rework `ProvenanceInterface` as discussed during backend design
    
    Add `ProvenanceResult` class to be returned by `content_find_first` and
    `content_find_all` methods. Rename some methods. Improve type annotations.

commit 2fd3f56b57f8db6691ae6b8b7cb7ac557b764172
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:31:16 2021 +0200

    Add tests for history graph topology

commit d45d6ff9e9317ecfe38d584df7297c548b654d28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:10:38 2021 +0200

    Fix database queries related to the origin-revision layer
    
    This required allowing null dates in the `revision` table so that revision can be added
    by the origin-revision layer algorithm but not recognized as already processed by the
    revision-content layer. Revision and origin entries are now inserted in the database
    prior to inserting rows to revision_in_origin and revision_before_revision relations,
    so that internal ids are properly resolved.

commit 0e2a3c64ce3c368b53c101c541e8aebcde789477
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jun 25 13:38:26 2021 +0200

    Add test to compare both `ArchiveInterface` implementations
    
    Improve documentation of the interface and complete pending TODO's.

commit 98bba93cccece2b47ec4cd5887997cb5bede1e87
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:25:15 2021 +0200

    Rename test files to keep naming convension
    
    Also added missing .msgpack file dump for new with-merges repository.

commit fa9198afb71bcf3b8abea07d88d763a430f7358e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:05:24 2021 +0200

    Refactor `ArchiveInterface` to fit origin-revision layer needs
    
    Replace `revision_get` method by `revision_get_parents` returning an iterable of
    parents' ids only, instead of a swh.model.model.Revision object.

commit 9e0c1aa099073887206c9334e17b49ee31bbef9a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 20:00:40 2021 +0200

    Use `Sha1Git` type to explicitly state the kind of identifiers
    
    Previous occurrences of `bytes` and `Sha1` are now correctly using `Sha1Git`.
    Also, some bytes conversion methods were replaced by their counterparts in
    the swh.model.hashutil module.

commit a27ffff67b6b14bf37d153bb9b1d1c2ae63773fc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 19:12:06 2021 +0200

    Add support for sha1 identifiers for origins

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/220/ for more details.

Harbormaster completed remote builds in B22377: Diff 21398.Jul 1 2021, 3:13 PM

aeviso added a child revision: D5957: Remove `db_utilis` in favour of `swh.core.db.BaseDb` methods.Jul 1 2021, 5:26 PM

rebase

Build is green

Patch application report for D5948 (id=21427)

Could not rebase; Attempt merge onto d892b29e40...

Updating d892b29..2a1aeb0
Fast-forward
 swh/provenance/__init__.py                         |  47 ++-
 swh/provenance/archive.py                          |  25 +-
 swh/provenance/backend.py                          | 322 +++++++++++++++++
 swh/provenance/cli.py                              |  28 +-
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/model.py                            |  53 ++-
 swh/provenance/origin.py                           |  21 +-
 swh/provenance/postgresql/archive.py               | 126 +++----
 swh/provenance/postgresql/provenancedb_base.py     | 402 ++++++++++++---------
 .../postgresql/provenancedb_with_path.py           | 117 +++---
 .../postgresql/provenancedb_without_path.py        |  96 ++---
 swh/provenance/provenance.py                       | 349 +++++++++---------
 swh/provenance/revision.py                         |  13 +-
 swh/provenance/sql/30-schema.sql                   |  30 +-
 swh/provenance/storage/archive.py                  |  39 +-
 swh/provenance/tests/conftest.py                   |  32 +-
 .../tests/data/generate_storage_from_git.py        |   3 +-
 .../data/history_graphs_with-merges_visits-01.yaml |  55 +++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 0 -> 7501 bytes
 ...repo_with_merges.yaml => with-merges_repo.yaml} |   0
 ...s-visits-01.yaml => with-merges_visits-01.yaml} |   0
 swh/provenance/tests/test_archive_interface.py     |  51 +++
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |  62 ++++
 swh/provenance/tests/test_origin_iterator.py       |   8 +-
 swh/provenance/tests/test_provenance_db.py         |   4 +-
 swh/provenance/tests/test_provenance_heuristics.py |  56 +--
 27 files changed, 1201 insertions(+), 744 deletions(-)
 create mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
 create mode 100644 swh/provenance/tests/data/with-merges.msgpack
 rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
 rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
 create mode 100644 swh/provenance/tests/test_archive_interface.py
 create mode 100644 swh/provenance/tests/test_history_graph.py

Changes applied before test

commit 2a1aeb023627ed31ee4eea617790607e1fa1ba68
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jun 29 14:28:54 2021 +0200

    Force `snapshot_get_heads` to return revisions in chronological order

commit f819e4332df40b1ef35ff737f2558de570379473
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 19:21:58 2021 +0200

    Add `ProvenanceStorageInterface` as discussed during backend design
    
    Rework backend-related classes to properly use the new interface.
    Adapt tests to the new structure as well.

commit 3672235c3258cf93fb37a82d060bf40ba1761b8b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:37:50 2021 +0200

    Move `ProvenanceBackend` implementation to a separate file

commit 6f4da6fed7e663273627ad4a46c8489ef0a0e784
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 1 13:47:26 2021 +0200

    Use `RealDictCursor` in `ProvenanceDBBase`
    
    to improve the way `ProvenanceResult`s are generated.

commit 07a30e43a76e170ab03764035da68dcf7db1fc3b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:28:32 2021 +0200

    Rework `ProvenanceInterface` as discussed during backend design
    
    Add `ProvenanceResult` class to be returned by `content_find_first` and
    `content_find_all` methods. Rename some methods. Improve type annotations.

commit 2fd3f56b57f8db6691ae6b8b7cb7ac557b764172
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:31:16 2021 +0200

    Add tests for history graph topology

commit d45d6ff9e9317ecfe38d584df7297c548b654d28
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:10:38 2021 +0200

    Fix database queries related to the origin-revision layer
    
    This required allowing null dates in the `revision` table so that revision can be added
    by the origin-revision layer algorithm but not recognized as already processed by the
    revision-content layer. Revision and origin entries are now inserted in the database
    prior to inserting rows to revision_in_origin and revision_before_revision relations,
    so that internal ids are properly resolved.

commit 0e2a3c64ce3c368b53c101c541e8aebcde789477
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jun 25 13:38:26 2021 +0200

    Add test to compare both `ArchiveInterface` implementations
    
    Improve documentation of the interface and complete pending TODO's.

commit 98bba93cccece2b47ec4cd5887997cb5bede1e87
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:25:15 2021 +0200

    Rename test files to keep naming convension
    
    Also added missing .msgpack file dump for new with-merges repository.

commit fa9198afb71bcf3b8abea07d88d763a430f7358e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:05:24 2021 +0200

    Refactor `ArchiveInterface` to fit origin-revision layer needs
    
    Replace `revision_get` method by `revision_get_parents` returning an iterable of
    parents' ids only, instead of a swh.model.model.Revision object.

commit 9e0c1aa099073887206c9334e17b49ee31bbef9a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 20:00:40 2021 +0200

    Use `Sha1Git` type to explicitly state the kind of identifiers
    
    Previous occurrences of `bytes` and `Sha1` are now correctly using `Sha1Git`.
    Also, some bytes conversion methods were replaced by their counterparts in
    the swh.model.hashutil module.

commit a27ffff67b6b14bf37d153bb9b1d1c2ae63773fc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 19:12:06 2021 +0200

    Add support for sha1 identifiers for origins

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/231/ for more details.

Harbormaster completed remote builds in B22405: Diff 21427.Jul 2 2021, 11:40 AM

rebase

Build is green

Patch application report for D5948 (id=21447)

Could not rebase; Attempt merge onto d892b29e40...

Updating d892b29..c72b2e2
Fast-forward
 swh/provenance/__init__.py                         |  47 ++-
 swh/provenance/archive.py                          |  25 +-
 swh/provenance/backend.py                          | 324 +++++++++++++++++
 swh/provenance/cli.py                              |  28 +-
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/model.py                            |  53 ++-
 swh/provenance/origin.py                           |  21 +-
 swh/provenance/postgresql/archive.py               | 126 +++----
 swh/provenance/postgresql/provenancedb_base.py     | 402 ++++++++++++---------
 .../postgresql/provenancedb_with_path.py           | 117 +++---
 .../postgresql/provenancedb_without_path.py        |  96 ++---
 swh/provenance/provenance.py                       | 349 +++++++++---------
 swh/provenance/revision.py                         |  13 +-
 swh/provenance/sql/30-schema.sql                   |  30 +-
 swh/provenance/storage/archive.py                  |  39 +-
 swh/provenance/tests/conftest.py                   |  32 +-
 .../tests/data/generate_storage_from_git.py        |   3 +-
 .../data/history_graphs_with-merges_visits-01.yaml |  55 +++
 swh/provenance/tests/data/with-merges.msgpack      | Bin 0 -> 7501 bytes
 ...repo_with_merges.yaml => with-merges_repo.yaml} |   0
 ...s-visits-01.yaml => with-merges_visits-01.yaml} |   0
 swh/provenance/tests/test_archive_interface.py     |  51 +++
 swh/provenance/tests/test_conftest.py              |   2 +-
 swh/provenance/tests/test_history_graph.py         |  62 ++++
 swh/provenance/tests/test_origin_iterator.py       |   8 +-
 swh/provenance/tests/test_provenance_db.py         |   4 +-
 swh/provenance/tests/test_provenance_heuristics.py |  56 +--
 27 files changed, 1203 insertions(+), 744 deletions(-)
 create mode 100644 swh/provenance/backend.py
 create mode 100644 swh/provenance/tests/data/history_graphs_with-merges_visits-01.yaml
 create mode 100644 swh/provenance/tests/data/with-merges.msgpack
 rename swh/provenance/tests/data/{repo_with_merges.yaml => with-merges_repo.yaml} (100%)
 rename swh/provenance/tests/data/{repo_with_merges-visits-01.yaml => with-merges_visits-01.yaml} (100%)
 create mode 100644 swh/provenance/tests/test_archive_interface.py
 create mode 100644 swh/provenance/tests/test_history_graph.py

Changes applied before test

commit c72b2e2428fe410c17220648d378c505bba35350
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Jun 29 14:28:54 2021 +0200

    Force `snapshot_get_heads` to return revisions in chronological order

commit 799839120cb99f22ce4272468ae0e388c335fb06
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 19:21:58 2021 +0200

    Add `ProvenanceStorageInterface` as discussed during backend design
    
    Rework backend-related classes to properly use the new interface.
    Adapt tests to the new structure as well.

commit 7c0a091ce5ffbf0a02dbe9d7fc84435ddd46cde2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:37:50 2021 +0200

    Move `ProvenanceBackend` implementation to a separate file

commit 34898ad3cb18c24a7d7bef79dcfe470c3a1374ef
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jul 1 13:47:26 2021 +0200

    Use `RealDictCursor` in `ProvenanceDBBase`
    
    to improve the way `ProvenanceResult`s are generated.

commit 721354c436b5f5a861800b11e6151afa1aa634b6
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Jun 28 14:28:32 2021 +0200

    Rework `ProvenanceInterface` as discussed during backend design
    
    Add `ProvenanceResult` class to be returned by `content_find_first` and
    `content_find_all` methods. Rename some methods. Improve type annotations.

commit 01f8d40ffccbcab6ecec6c2cf85478364e006caa
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:31:16 2021 +0200

    Add tests for history graph topology

commit b7fdcdec7ea96101d62a57d9aeed114c897df961
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:10:38 2021 +0200

    Fix database queries related to the origin-revision layer
    
    This required allowing null dates in the `revision` table so that revision can be added
    by the origin-revision layer algorithm but not recognized as already processed by the
    revision-content layer. Revision and origin entries are now inserted in the database
    prior to inserting rows to revision_in_origin and revision_before_revision relations,
    so that internal ids are properly resolved.

commit 0e2a3c64ce3c368b53c101c541e8aebcde789477
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Jun 25 13:38:26 2021 +0200

    Add test to compare both `ArchiveInterface` implementations
    
    Improve documentation of the interface and complete pending TODO's.

commit 98bba93cccece2b47ec4cd5887997cb5bede1e87
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:25:15 2021 +0200

    Rename test files to keep naming convension
    
    Also added missing .msgpack file dump for new with-merges repository.

commit fa9198afb71bcf3b8abea07d88d763a430f7358e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Jun 24 16:05:24 2021 +0200

    Refactor `ArchiveInterface` to fit origin-revision layer needs
    
    Replace `revision_get` method by `revision_get_parents` returning an iterable of
    parents' ids only, instead of a swh.model.model.Revision object.

commit 9e0c1aa099073887206c9334e17b49ee31bbef9a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 20:00:40 2021 +0200

    Use `Sha1Git` type to explicitly state the kind of identifiers
    
    Previous occurrences of `bytes` and `Sha1` are now correctly using `Sha1Git`.
    Also, some bytes conversion methods were replaced by their counterparts in
    the swh.model.hashutil module.

commit a27ffff67b6b14bf37d153bb9b1d1c2ae63773fc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Jun 23 19:12:06 2021 +0200

    Add support for sha1 identifiers for origins

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/245/ for more details.

Harbormaster completed remote builds in B22425: Diff 21447.Jul 2 2021, 4:39 PM

Closed by commit rDPROVc72b2e2428fe: Force `snapshot_get_heads` to return revisions in chronological order (authored by aeviso). · Explain WhyJul 2 2021, 5:24 PM

This revision was automatically updated to reflect the committed changes.

aeviso added a commit: rDPROVc72b2e2428fe: Force `snapshot_get_heads` to return revisions in chronological order.

Force `snapshot_get_heads` to return revisions in chronological order
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Patch application report for D5948 (id=21349)

Changes applied before test

Patch application report for D5948 (id=21354)

Changes applied before test

Patch application report for D5948 (id=21365)

Changes applied before test

Patch application report for D5948 (id=21371)

Changes applied before test

Patch application report for D5948 (id=21387)

Changes applied before test

Patch application report for D5948 (id=21388)

Changes applied before test

Patch application report for D5948 (id=21389)

Changes applied before test

Patch application report for D5948 (id=21398)

Changes applied before test

Patch application report for D5948 (id=21427)

Changes applied before test

Patch application report for D5948 (id=21447)

Changes applied before test

Revision Contents
Changeset List

Diff 21447

swh/provenance/archive.py

swh/provenance/postgresql/archive.py

swh/provenance/storage/archive.py

Force `snapshot_get_heads` to return revisions in chronological orderClosedPublicActions

Details

Diff Detail

Event Timeline

Patch application report for D5948 (id=21349)

Changes applied before test

Patch application report for D5948 (id=21354)

Changes applied before test

Patch application report for D5948 (id=21365)

Changes applied before test

Patch application report for D5948 (id=21371)

Changes applied before test

Patch application report for D5948 (id=21387)

Changes applied before test

Patch application report for D5948 (id=21388)

Changes applied before test

Patch application report for D5948 (id=21389)

Changes applied before test

Patch application report for D5948 (id=21398)

Changes applied before test

Patch application report for D5948 (id=21427)

Changes applied before test

Patch application report for D5948 (id=21447)

Changes applied before test

Revision ContentsChangeset List

Diff 21447

swh/provenance/archive.py

swh/provenance/postgresql/archive.py

swh/provenance/storage/archive.py

Force `snapshot_get_heads` to return revisions in chronological order
ClosedPublic
Actions

Revision Contents
Changeset List