Page MenuHomeSoftware Heritage

Refactor `raise_on_commit` logic with a decorator
ClosedPublic

Authored by aeviso on Dec 2 2021, 5:20 PM.

Details

Diff Detail

Repository
rDPROV Provenance database
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D6734 (id=24459)

Could not rebase; Attempt merge onto dd1d7aa233...

Updating dd1d7aa..bb803e8
Fast-forward
 sql/upgrades/002.sql                               |  17 ++
 swh/provenance/api/serializers.py                  |   5 +-
 swh/provenance/api/server.py                       |  44 +++-
 swh/provenance/cli.py                              |  40 +++
 swh/provenance/directory.py                        |  86 +++++++
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/interface.py                        |  54 ++--
 swh/provenance/mongo/backend.py                    |  48 ++--
 swh/provenance/origin.py                           |   2 +-
 swh/provenance/postgresql/provenance.py            | 276 ++++++++++-----------
 swh/provenance/provenance.py                       |  70 +++++-
 swh/provenance/revision.py                         |  40 +--
 swh/provenance/sql/30-schema.sql                   |  71 +++---
 swh/provenance/tests/test_cli.py                   |   1 +
 swh/provenance/tests/test_conflict_resolution.py   |  43 ++--
 swh/provenance/tests/test_directory_flatten.py     |  72 ++++++
 swh/provenance/tests/test_directory_iterator.py    |  29 +++
 swh/provenance/tests/test_history_graph.py         |   2 +-
 swh/provenance/tests/test_isochrone_graph.py       |   2 +-
 swh/provenance/tests/test_provenance_storage.py    |  39 ++-
 .../tests/test_revision_content_layer.py           |  49 +++-
 21 files changed, 667 insertions(+), 327 deletions(-)
 create mode 100644 sql/upgrades/002.sql
 create mode 100644 swh/provenance/directory.py
 create mode 100644 swh/provenance/tests/test_directory_flatten.py
 create mode 100644 swh/provenance/tests/test_directory_iterator.py
Changes applied before test
commit bb803e8531228cb4fbb323ce01787f67f5ba2fb1
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Dec 2 17:19:39 2021 +0100

    Refactor `raise_on_commit` logic with a decorator

commit 262057262e85c5284ed2f4f03c53a7e9c77fa7a7
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 16:53:46 2021 +0100

    Add new flag to skip directory flattening while processing revisions

commit 5448b6ee5bc799c73cfe49d67c97768dadfbb8cc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 15:45:12 2021 +0100

    Add support to flatten directories in the isochrone frontiers separately
    
    Building on the previous commit, a new entry point is added to the module
    allowing to iterate over a list of directories that are already identified
    as isochrone frontiers in the provenance model, but no flat models for
    their content has been created yet. This iteration produces such flat
    models.

commit 812df71d99daacb25d1df73522cb754b0842af83
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 15:00:28 2021 +0100

    Unify parameter order between provenance and archive objects across the module

commit 765135807ee60342f0b9e62d584c5bd46fedb069
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 13:21:33 2021 +0100

    Add explicit flag for flattenned directories to `ProvenanceStorageInterface`
    
    Both contents and directories should always have an associated date in
    the storage. Flattening of a direcory is know explicitly acknowledged
    by setting the newly added flag.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/523/ for more details.

Build is green

Patch application report for D6734 (id=24461)

Could not rebase; Attempt merge onto dd1d7aa233...

Updating dd1d7aa..5dbeea1
Fast-forward
 sql/upgrades/002.sql                               |  17 ++
 swh/provenance/api/serializers.py                  |   5 +-
 swh/provenance/api/server.py                       |  44 +++-
 swh/provenance/cli.py                              |  40 +++
 swh/provenance/directory.py                        |  86 +++++++
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/interface.py                        |  54 ++--
 swh/provenance/mongo/backend.py                    |  48 ++--
 swh/provenance/origin.py                           |   2 +-
 swh/provenance/postgresql/provenance.py            | 274 ++++++++++-----------
 swh/provenance/provenance.py                       |  70 +++++-
 swh/provenance/revision.py                         |  40 +--
 swh/provenance/sql/30-schema.sql                   |  71 +++---
 swh/provenance/tests/test_cli.py                   |   1 +
 swh/provenance/tests/test_conflict_resolution.py   |  43 ++--
 swh/provenance/tests/test_directory_flatten.py     |  72 ++++++
 swh/provenance/tests/test_directory_iterator.py    |  29 +++
 swh/provenance/tests/test_history_graph.py         |   2 +-
 swh/provenance/tests/test_isochrone_graph.py       |   2 +-
 swh/provenance/tests/test_provenance_storage.py    |  39 ++-
 .../tests/test_revision_content_layer.py           |  49 +++-
 21 files changed, 666 insertions(+), 326 deletions(-)
 create mode 100644 sql/upgrades/002.sql
 create mode 100644 swh/provenance/directory.py
 create mode 100644 swh/provenance/tests/test_directory_flatten.py
 create mode 100644 swh/provenance/tests/test_directory_iterator.py
Changes applied before test
commit 5dbeea1a1e6b96a527d1da8a276554aac672000b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Dec 2 17:19:39 2021 +0100

    Refactor `raise_on_commit` logic with a decorator

commit 262057262e85c5284ed2f4f03c53a7e9c77fa7a7
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 16:53:46 2021 +0100

    Add new flag to skip directory flattening while processing revisions

commit 5448b6ee5bc799c73cfe49d67c97768dadfbb8cc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 15:45:12 2021 +0100

    Add support to flatten directories in the isochrone frontiers separately
    
    Building on the previous commit, a new entry point is added to the module
    allowing to iterate over a list of directories that are already identified
    as isochrone frontiers in the provenance model, but no flat models for
    their content has been created yet. This iteration produces such flat
    models.

commit 812df71d99daacb25d1df73522cb754b0842af83
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 15:00:28 2021 +0100

    Unify parameter order between provenance and archive objects across the module

commit 765135807ee60342f0b9e62d584c5bd46fedb069
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 13:21:33 2021 +0100

    Add explicit flag for flattenned directories to `ProvenanceStorageInterface`
    
    Both contents and directories should always have an associated date in
    the storage. Flattening of a direcory is know explicitly acknowledged
    by setting the newly added flag.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/524/ for more details.

aeviso requested review of this revision.Dec 2 2021, 5:29 PM

Build is green

Patch application report for D6734 (id=24463)

Could not rebase; Attempt merge onto dd1d7aa233...

Updating dd1d7aa..c348cf8
Fast-forward
 sql/upgrades/002.sql                               |  17 ++
 swh/provenance/api/serializers.py                  |   5 +-
 swh/provenance/api/server.py                       |  44 +++-
 swh/provenance/cli.py                              |  40 +++
 swh/provenance/directory.py                        |  86 +++++++
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/interface.py                        |  54 ++--
 swh/provenance/mongo/backend.py                    |  48 ++--
 swh/provenance/origin.py                           |   2 +-
 swh/provenance/postgresql/provenance.py            | 274 ++++++++++-----------
 swh/provenance/provenance.py                       |  70 +++++-
 swh/provenance/revision.py                         |  40 +--
 swh/provenance/sql/30-schema.sql                   |  71 +++---
 swh/provenance/tests/test_cli.py                   |   1 +
 swh/provenance/tests/test_conflict_resolution.py   |  43 ++--
 swh/provenance/tests/test_directory_flatten.py     |  72 ++++++
 swh/provenance/tests/test_directory_iterator.py    |  29 +++
 swh/provenance/tests/test_history_graph.py         |   2 +-
 swh/provenance/tests/test_isochrone_graph.py       |   2 +-
 swh/provenance/tests/test_provenance_storage.py    |  39 ++-
 .../tests/test_revision_content_layer.py           |  49 +++-
 21 files changed, 666 insertions(+), 326 deletions(-)
 create mode 100644 sql/upgrades/002.sql
 create mode 100644 swh/provenance/directory.py
 create mode 100644 swh/provenance/tests/test_directory_flatten.py
 create mode 100644 swh/provenance/tests/test_directory_iterator.py
Changes applied before test
commit c348cf806ad4a205c0bf1e24f6a4bdd265796fe2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Dec 2 17:19:39 2021 +0100

    Refactor `raise_on_commit` logic with a decorator

commit 262057262e85c5284ed2f4f03c53a7e9c77fa7a7
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 16:53:46 2021 +0100

    Add new flag to skip directory flattening while processing revisions

commit 5448b6ee5bc799c73cfe49d67c97768dadfbb8cc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 15:45:12 2021 +0100

    Add support to flatten directories in the isochrone frontiers separately
    
    Building on the previous commit, a new entry point is added to the module
    allowing to iterate over a list of directories that are already identified
    as isochrone frontiers in the provenance model, but no flat models for
    their content has been created yet. This iteration produces such flat
    models.

commit 812df71d99daacb25d1df73522cb754b0842af83
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 15:00:28 2021 +0100

    Unify parameter order between provenance and archive objects across the module

commit 765135807ee60342f0b9e62d584c5bd46fedb069
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 13:21:33 2021 +0100

    Add explicit flag for flattenned directories to `ProvenanceStorageInterface`
    
    Both contents and directories should always have an associated date in
    the storage. Flattening of a direcory is know explicitly acknowledged
    by setting the newly added flag.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/525/ for more details.

Build is green

Patch application report for D6734 (id=24466)

Could not rebase; Attempt merge onto dd1d7aa233...

Updating dd1d7aa..f7ea16a
Fast-forward
 sql/upgrades/002.sql                               |  17 ++
 swh/provenance/api/serializers.py                  |   5 +-
 swh/provenance/api/server.py                       |  44 +++-
 swh/provenance/cli.py                              |  40 +++
 swh/provenance/directory.py                        |  86 +++++++
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/interface.py                        |  54 ++--
 swh/provenance/mongo/backend.py                    |  48 ++--
 swh/provenance/origin.py                           |   2 +-
 swh/provenance/postgresql/provenance.py            | 274 ++++++++++-----------
 swh/provenance/provenance.py                       |  70 +++++-
 swh/provenance/revision.py                         |  40 +--
 swh/provenance/sql/30-schema.sql                   |  71 +++---
 swh/provenance/tests/test_cli.py                   |   1 +
 swh/provenance/tests/test_conflict_resolution.py   |  43 ++--
 swh/provenance/tests/test_directory_flatten.py     |  72 ++++++
 swh/provenance/tests/test_directory_iterator.py    |  29 +++
 swh/provenance/tests/test_history_graph.py         |   2 +-
 swh/provenance/tests/test_isochrone_graph.py       |   2 +-
 swh/provenance/tests/test_provenance_storage.py    |  39 ++-
 .../tests/test_revision_content_layer.py           |  51 +++-
 21 files changed, 668 insertions(+), 326 deletions(-)
 create mode 100644 sql/upgrades/002.sql
 create mode 100644 swh/provenance/directory.py
 create mode 100644 swh/provenance/tests/test_directory_flatten.py
 create mode 100644 swh/provenance/tests/test_directory_iterator.py
Changes applied before test
commit f7ea16a592c024de1de605f004fc9afc4d5a0f0c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Dec 2 17:19:39 2021 +0100

    Refactor `raise_on_commit` logic with a decorator

commit 7b4b3f24b274b64840ee1f050926a113b860137f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 16:53:46 2021 +0100

    Add new flag to skip directory flattening while processing revisions

commit 5448b6ee5bc799c73cfe49d67c97768dadfbb8cc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 15:45:12 2021 +0100

    Add support to flatten directories in the isochrone frontiers separately
    
    Building on the previous commit, a new entry point is added to the module
    allowing to iterate over a list of directories that are already identified
    as isochrone frontiers in the provenance model, but no flat models for
    their content has been created yet. This iteration produces such flat
    models.

commit 812df71d99daacb25d1df73522cb754b0842af83
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 15:00:28 2021 +0100

    Unify parameter order between provenance and archive objects across the module

commit 765135807ee60342f0b9e62d584c5bd46fedb069
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 13:21:33 2021 +0100

    Add explicit flag for flattenned directories to `ProvenanceStorageInterface`
    
    Both contents and directories should always have an associated date in
    the storage. Flattening of a direcory is know explicitly acknowledged
    by setting the newly added flag.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/527/ for more details.

This revision is now accepted and ready to land.Dec 3 2021, 9:56 AM

Build is green

Patch application report for D6734 (id=24722)

Could not rebase; Attempt merge onto dd1d7aa233...

Updating dd1d7aa..78b8b77
Fast-forward
 sql/upgrades/002.sql                               |  17 ++
 swh/provenance/api/serializers.py                  |   5 +-
 swh/provenance/api/server.py                       |  44 +++-
 swh/provenance/cli.py                              | 111 ++++++++-
 swh/provenance/directory.py                        |  86 +++++++
 swh/provenance/graph.py                            |   4 +-
 swh/provenance/interface.py                        |  54 ++--
 swh/provenance/mongo/backend.py                    |  48 ++--
 swh/provenance/origin.py                           |   2 +-
 swh/provenance/postgresql/provenance.py            | 274 ++++++++++-----------
 swh/provenance/provenance.py                       |  72 +++++-
 swh/provenance/revision.py                         |  40 +--
 swh/provenance/sql/30-schema.sql                   |  71 +++---
 swh/provenance/tests/test_cli.py                   |   1 +
 swh/provenance/tests/test_conflict_resolution.py   |  43 ++--
 swh/provenance/tests/test_directory_flatten.py     |  72 ++++++
 swh/provenance/tests/test_directory_iterator.py    |  29 +++
 swh/provenance/tests/test_history_graph.py         |   2 +-
 swh/provenance/tests/test_isochrone_graph.py       |   2 +-
 swh/provenance/tests/test_provenance_storage.py    |  39 ++-
 .../tests/test_revision_content_layer.py           |  51 +++-
 21 files changed, 733 insertions(+), 334 deletions(-)
 create mode 100644 sql/upgrades/002.sql
 create mode 100644 swh/provenance/directory.py
 create mode 100644 swh/provenance/tests/test_directory_flatten.py
 create mode 100644 swh/provenance/tests/test_directory_iterator.py
Changes applied before test
commit 78b8b77cdaaa302e140df25e9c98f0a25dfe3278
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Dec 2 17:19:39 2021 +0100

    Refactor `raise_on_commit` logic with a decorator

commit 5a86c235de7b8c1b74aed370a600ade36c3412f6
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 16:53:46 2021 +0100

    Add new flag to skip directory flattening while processing revisions

commit 0f2025f6ef454616537103fa720479987cba1278
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 15:45:12 2021 +0100

    Add support to flatten directories in the isochrone frontiers separately
    
    Building on the previous commit, a new entry point is added to the module
    allowing to iterate over a list of directories that are already identified
    as isochrone frontiers in the provenance model, but no flat models for
    their content has been created yet. This iteration produces such flat
    models.

commit 052e25da505c77da90d1c54ce0ade775117422e4
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 15:00:28 2021 +0100

    Unify parameter order between provenance and archive objects across the module

commit f4f48923e86ef0054642165bcb9ecf4387d70bb8
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Dec 1 13:21:33 2021 +0100

    Add explicit flag for flattenned directories to `ProvenanceStorageInterface`
    
    Both contents and directories should always have an associated date in the storage.
    Flattening of a directory is now explicitly acknowledged by setting the newly added
    flag. The idea is to allow to postpone the creation of flat models for directories
    in the isochrone frontier (the algorithm will be refactored in the commits to come).

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/539/ for more details.