Updating 4c087ea..7c9e557
Fast-forward
 .gitignore                                      |   4 +-
 mypy.ini                                        |   3 +
 pytest.ini                                      |   4 +
 requirements-test.txt                           |   2 +-
 requirements.txt                                |   3 +-
 swh/provenance/__init__.py                      |  28 +-
 swh/provenance/api/client.py                    | 557 +++++++++++++++-
 swh/provenance/api/server.py                    | 846 +++++++++++++++++++++---
 swh/provenance/cli.py                           |  97 +--
 swh/provenance/graph.py                         |   9 +
 swh/provenance/interface.py                     |  47 +-
 swh/provenance/model.py                         |   6 +-
 swh/provenance/mongo/backend.py                 |  61 +-
 swh/provenance/origin.py                        |  17 +-
 swh/provenance/postgresql/archive.py            |  15 +-
 swh/provenance/postgresql/provenance.py         |  58 +-
 swh/provenance/provenance.py                    | 165 +++--
 swh/provenance/revision.py                      |  31 +-
 swh/provenance/storage/archive.py               |  15 +-
 swh/provenance/tests/conftest.py                |  81 +--
 swh/provenance/tests/test_provenance_storage.py |  21 +-
 tox.ini                                         |   3 +-
 22 files changed, 1740 insertions(+), 333 deletions(-)

Changes applied before test

commit 7c9e55742e536b56ea9b7e40b4930fb91d5b762f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Oct 5 12:01:25 2021 +0200

    Remove direct dependencies on deprecated `swh.model.identifiers` module

commit 377e0ea2e63cb15e0054728c7401d2f6136dea6e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 21 16:13:53 2021 +0200

    Add support for remote backend on existing storage tests

commit cd4056be39e8152276cc5d65f39d512021714d84
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add StatsD support to client to be compliant with the other provenance
    storage implementations

commit 2eaf7200e8a97e42b291313f04cf24ae6c6ce9f2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit d1913496c81be2fae1eff1cf93a17c8439991706
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 04ff73ea98f5f239cee6a126c75767f4617e330c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:59:38 2021 +0200

    Make old StatsD metrics style compliant with the rest of the module

commit 1bd6b22aae6a356e18f65005fc7e1c162e6f38c6
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 14:08:10 2021 +0200

    Add StatsD support to graph submodule
    
    Time stats of graphs creation and counter of amount of invalidated isochrone frontiers

commit 1ad78362fb415ea1d88a1d416da9991896e68d43
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 13:53:57 2021 +0200

    Add StatsD support to provenance storage implementations

commit e2a1843d5ebe01a9cdfe46b6b74dde1e293b8c01
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:21:42 2021 +0200

    Add StatsD support to provenance backend

commit 246e55f9b7e3475ea4509e08370827a3190db916
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:17:34 2021 +0200

    Split `Provenance::flush` method in two (one per layer)

commit f0210c3753c3a4122ee3c54f7fac97d170a142fa
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `open`/`close` methods to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    This allows to have an explicit mechanism to allocate/release resources when needed.
    The necessary methods for the classes implementing these interfaces to be turned in contexts
    managers are added as well (ie. `__enter__`/`__exit__`).

commit 172e327c25883bee768a9c16b850ce6aab7e2eb2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Remove remote provenance storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/430/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/430/console

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 5 2021, 12:24 PM

Harbormaster failed remote builds in B24216: Diff 23303!

rebase

Build is green

Patch application report for D6409 (id=23314)

Could not rebase; Attempt merge onto 04ff73ea98...

Updating 04ff73e..a169030
Fast-forward
 .gitignore                                      |   4 +-
 mypy.ini                                        |   3 +
 pytest.ini                                      |   2 +
 requirements-test.txt                           |   1 +
 requirements.txt                                |   1 +
 swh/provenance/__init__.py                      |   8 +
 swh/provenance/api/client.py                    | 564 +++++++++++++++++
 swh/provenance/api/server.py                    | 786 +++++++++++++++++++++++-
 swh/provenance/cli.py                           |  26 +-
 swh/provenance/model.py                         |   6 +-
 swh/provenance/tests/conftest.py                |  24 +-
 swh/provenance/tests/test_provenance_storage.py |  21 +-
 tox.ini                                         |   3 +-
 13 files changed, 1420 insertions(+), 29 deletions(-)

Changes applied before test

commit a169030e8e64d35c8f3c2cd053b7ee6dfa5629e6
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Oct 5 12:01:25 2021 +0200

    Remove direct dependencies on deprecated `swh.model.identifiers` module

commit f1842b95799b6816e760f44086ab166e17c98321
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 21 16:13:53 2021 +0200

    Add support for remote backend on existing storage tests

commit cd4056be39e8152276cc5d65f39d512021714d84
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add StatsD support to client to be compliant with the other provenance
    storage implementations

commit 2eaf7200e8a97e42b291313f04cf24ae6c6ce9f2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit d1913496c81be2fae1eff1cf93a17c8439991706
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/432/ for more details.

Harbormaster completed remote builds in B24227: Diff 23314.Oct 5 2021, 2:39 PM

aeviso requested review of this revision.Oct 5 2021, 2:39 PM

Thanks!

Sorry, I meant to do this myself, but didn't get around it yet.

This revision is now accepted and ready to land.Oct 5 2021, 2:57 PM

In D6409#166428, @vlorentz wrote:

Thanks!

Sorry, I meant to do this myself, but didn't get around it yet.

If you run the tests you'll notice that the deprecation warning still appears. I guess this has to do with some indirect dependency on swh.model.identifiers still being present.

yes, that's my guess too

rebase

aeviso removed a parent revision: D6339: Add support for remote backend on existing storage tests.Oct 11 2021, 5:32 PM

aeviso added a child revision: D6456: Move path normalization function to `util` submodule.Oct 11 2021, 5:33 PM

Build is green

Patch application report for D6409 (id=23451)

Rebasing onto 04ff73ea98...

Current branch diff-target is up to date.

Changes applied before test

commit 2c9ef5673b369f2baa83b11ec9256c6aafc3a855
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Oct 5 12:01:25 2021 +0200

    Remove direct dependencies on deprecated `swh.model.identifiers` module

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/434/ for more details.

Harbormaster completed remote builds in B24367: Diff 23451.Oct 11 2021, 5:34 PM

Closed by commit rDPROV2c9ef5673b36: Remove direct dependencies on deprecated `swh.model.identifiers` module (authored by aeviso). · Explain WhyOct 11 2021, 5:38 PM

This revision was automatically updated to reflect the committed changes.

aeviso added a commit: rDPROV2c9ef5673b36: Remove direct dependencies on deprecated `swh.model.identifiers` module.

aeviso removed a child revision: D6456: Move path normalization function to `util` submodule.Oct 11 2021, 5:39 PM