Page MenuHomeSoftware Heritage
Differential D6358

Make old StatsD metrics style compliant with the rest of the module
ClosedPublic
Actions

Authored by aeviso on Sep 27 2021, 4:04 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Commits
rDPROV04ff73ea98f5: Make old StatsD metrics style compliant with the rest of the module
Summary

Depends on D6353.

Diff Detail

Event Timeline

aeviso created this revision.Sep 27 2021, 4:04 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptSep 27 2021, 4:04 PM
swh-public-ci added a comment.Sep 27 2021, 4:12 PM

Build is green

Patch application report for D6358 (id=23097)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..055001a
Fast-forward
 .gitignore                              |   4 +-
 mypy.ini                                |   3 +
 pytest.ini                              |   5 +
 requirements-test.txt                   |   2 +-
 swh/provenance/__init__.py              |  14 +-
 swh/provenance/api/client.py            | 537 +++++++++++++++++++-
 swh/provenance/api/server.py            | 844 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  33 +-
 swh/provenance/graph.py                 |   9 +
 swh/provenance/interface.py             |   9 +
 swh/provenance/mongo/backend.py         |  24 +
 swh/provenance/origin.py                |  17 +-
 swh/provenance/postgresql/archive.py    |  15 +-
 swh/provenance/postgresql/provenance.py |  24 +
 swh/provenance/provenance.py            | 151 +++---
 swh/provenance/revision.py              |  31 +-
 swh/provenance/storage/archive.py       |  15 +-
 swh/provenance/tests/conftest.py        |  69 +--
 tox.ini                                 |   3 +-
 19 files changed, 1561 insertions(+), 248 deletions(-)
Changes applied before test
commit 055001ace68b0ed5dcd53801d0954b332dfcfbea
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:59:38 2021 +0200

    Make old StatsD metrics style compliant with the rest of the module

commit 9bb53b94b60619f79736724d4174a7baf8e26e9f
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 14:08:10 2021 +0200

    Add StatsD support to graph submodule
    
    Time stats of graphs creation and counter of amount of invalidated isochrone frontiers

commit a2d475e9ea94392a796b106a879c01bbcbbb5dcf
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 13:53:57 2021 +0200

    Add StatsD support to all three storage implemetations

commit 65d40bd64f74983ccef997f25e696baeb5d78173
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:21:42 2021 +0200

    Add support for StatsD to provenance backend

commit eb28c1d07615b602660c53bdab3e04b01ee8289a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:17:34 2021 +0200

    Split `Provenance::flush` method in two (one per layer)
    
    Fix preferred origin setting on origin-revision layer.

commit 4995134725cb5e62bcdf1d0c19ea4e2fabac5465
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.
    The new `ProvenanceStorageRabbitMQ` remote storage should be used instead.

commit 50a1441f2068a2abe1293be893e66806cf0977b9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 21 16:13:53 2021 +0200

    Add support for remote backend on existing storage tests

commit ba22e90f707688aed716d20253256ee12c414a33
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling

commit 4ec6d3359e20920469cde43516bc120d4352f915
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit f6f174fb76b2c42435b6c75eb90c175d9cb0fca5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 437c2b4ad60e15c0b102988a99766b2c503c91d5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/384/ for more details.

Harbormaster completed remote builds in B24019: Diff 23097.Sep 27 2021, 4:12 PM
aeviso requested review of this revision.Sep 27 2021, 4:12 PM
aeviso updated this revision to Diff 23102.Sep 27 2021, 4:14 PM

rebase

swh-public-ci added a comment.Sep 27 2021, 4:22 PM

Build is green

Patch application report for D6358 (id=23102)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..c1204de
Fast-forward
 .gitignore                              |   4 +-
 mypy.ini                                |   3 +
 pytest.ini                              |   5 +
 requirements-test.txt                   |   2 +-
 swh/provenance/__init__.py              |  14 +-
 swh/provenance/api/client.py            | 537 +++++++++++++++++++-
 swh/provenance/api/server.py            | 844 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  33 +-
 swh/provenance/graph.py                 |   9 +
 swh/provenance/interface.py             |   9 +
 swh/provenance/mongo/backend.py         |  24 +
 swh/provenance/origin.py                |  17 +-
 swh/provenance/postgresql/archive.py    |  15 +-
 swh/provenance/postgresql/provenance.py |  24 +
 swh/provenance/provenance.py            | 151 +++---
 swh/provenance/revision.py              |  31 +-
 swh/provenance/storage/archive.py       |  15 +-
 swh/provenance/tests/conftest.py        |  69 +--
 tox.ini                                 |   3 +-
 19 files changed, 1561 insertions(+), 248 deletions(-)
Changes applied before test
commit c1204de96565151349fb86d91409c55b066af5f7
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:59:38 2021 +0200

    Make old StatsD metrics style compliant with the rest of the module

commit ec09c1035d95df05e12470e4e8d551dd464ee363
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 14:08:10 2021 +0200

    Add StatsD support to graph submodule
    
    Time stats of graphs creation and counter of amount of invalidated isochrone frontiers

commit cdcbfe4b7fae095cb04a863b619b5eb865689601
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 13:53:57 2021 +0200

    Add StatsD support to all three storage implementations

commit f93689681cce625e18ef91040ba6113074142a5d
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:21:42 2021 +0200

    Add StatsD support to provenance backend

commit eb28c1d07615b602660c53bdab3e04b01ee8289a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:17:34 2021 +0200

    Split `Provenance::flush` method in two (one per layer)
    
    Fix preferred origin setting on origin-revision layer.

commit 4995134725cb5e62bcdf1d0c19ea4e2fabac5465
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.
    The new `ProvenanceStorageRabbitMQ` remote storage should be used instead.

commit 50a1441f2068a2abe1293be893e66806cf0977b9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 21 16:13:53 2021 +0200

    Add support for remote backend on existing storage tests

commit ba22e90f707688aed716d20253256ee12c414a33
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling

commit 4ec6d3359e20920469cde43516bc120d4352f915
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit f6f174fb76b2c42435b6c75eb90c175d9cb0fca5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 437c2b4ad60e15c0b102988a99766b2c503c91d5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/388/ for more details.

Harbormaster completed remote builds in B24024: Diff 23102.Sep 27 2021, 4:22 PM
douardda added a subscriber: douardda.Sep 27 2021, 4:55 PM
douardda added inline comments.
swh/provenance/origin.py
48

I get this is not part of this diff, but why is this called "main" instead of "origin_add"?

The "method" tag should stick to the name of the decorated method/function.

aeviso marked an inline comment as done.Sep 27 2021, 5:04 PM
aeviso added inline comments.
swh/provenance/origin.py
48

This is the main method of the origin-revision layer algorithm. We identify the metrics associated to this layer with the metric tag. Is is done in the same way for the other algorithm

aeviso updated this revision to Diff 23114.Sep 28 2021, 10:11 AM
aeviso marked an inline comment as done.

rebase

swh-public-ci added a comment.Sep 28 2021, 10:19 AM

Build is green

Patch application report for D6358 (id=23114)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..6ec0945
Fast-forward
 .gitignore                              |   4 +-
 mypy.ini                                |   3 +
 pytest.ini                              |   5 +
 requirements-test.txt                   |   2 +-
 swh/provenance/__init__.py              |  14 +-
 swh/provenance/api/client.py            | 537 +++++++++++++++++++-
 swh/provenance/api/server.py            | 844 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  33 +-
 swh/provenance/graph.py                 |   9 +
 swh/provenance/interface.py             |   9 +
 swh/provenance/mongo/backend.py         |  24 +
 swh/provenance/origin.py                |  17 +-
 swh/provenance/postgresql/archive.py    |  15 +-
 swh/provenance/postgresql/provenance.py |  24 +
 swh/provenance/provenance.py            | 146 +++---
 swh/provenance/revision.py              |  31 +-
 swh/provenance/storage/archive.py       |  15 +-
 swh/provenance/tests/conftest.py        |  69 +--
 tox.ini                                 |   3 +-
 19 files changed, 1556 insertions(+), 248 deletions(-)
Changes applied before test
commit 6ec09456c388a8b3b8043d5755ab2badb523d51a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:59:38 2021 +0200

    Make old StatsD metrics style compliant with the rest of the module

commit f7dfda21345af28866749646174b1154043b89b8
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 14:08:10 2021 +0200

    Add StatsD support to graph submodule
    
    Time stats of graphs creation and counter of amount of invalidated isochrone frontiers

commit 0e566272a74fb8f843f5c2cb513e8b3e655c695a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 13:53:57 2021 +0200

    Add StatsD support to all three storage implementations

commit f1f4b9769b9d24e04b45713d729940bc3782785c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:21:42 2021 +0200

    Add StatsD support to provenance backend

commit dd1b27df175f6185fd2deed405b0325d9a09d0dd
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:17:34 2021 +0200

    Split `Provenance::flush` method in two (one per layer)

commit 4995134725cb5e62bcdf1d0c19ea4e2fabac5465
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.
    The new `ProvenanceStorageRabbitMQ` remote storage should be used instead.

commit 50a1441f2068a2abe1293be893e66806cf0977b9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 21 16:13:53 2021 +0200

    Add support for remote backend on existing storage tests

commit ba22e90f707688aed716d20253256ee12c414a33
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling

commit 4ec6d3359e20920469cde43516bc120d4352f915
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit f6f174fb76b2c42435b6c75eb90c175d9cb0fca5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 437c2b4ad60e15c0b102988a99766b2c503c91d5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/393/ for more details.

Harbormaster completed remote builds in B24031: Diff 23114.Sep 28 2021, 10:19 AM
aeviso updated this revision to Diff 23178.Sep 29 2021, 2:36 PM

rebase

swh-public-ci added a comment.Sep 29 2021, 2:47 PM

Build is green

Patch application report for D6358 (id=23178)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..59d86f6
Fast-forward
 .gitignore                              |   4 +-
 mypy.ini                                |   3 +
 pytest.ini                              |   5 +
 requirements-test.txt                   |   2 +-
 requirements.txt                        |   1 +
 swh/provenance/__init__.py              |  28 +-
 swh/provenance/api/client.py            | 539 +++++++++++++++++++-
 swh/provenance/api/server.py            | 844 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  39 +-
 swh/provenance/graph.py                 |   9 +
 swh/provenance/interface.py             |  20 +
 swh/provenance/mongo/backend.py         |  41 +-
 swh/provenance/origin.py                |  17 +-
 swh/provenance/postgresql/archive.py    |  15 +-
 swh/provenance/postgresql/provenance.py |  40 +-
 swh/provenance/provenance.py            | 149 +++---
 swh/provenance/revision.py              |  31 +-
 swh/provenance/storage/archive.py       |  15 +-
 swh/provenance/tests/conftest.py        |  84 ++--
 tox.ini                                 |   3 +-
 20 files changed, 1618 insertions(+), 271 deletions(-)
Changes applied before test
commit 59d86f67b9a0c7100c07160e9deea8164d343a87
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:59:38 2021 +0200

    Make old StatsD metrics style compliant with the rest of the module

commit 34d7be2a50de621963f9220c8184ccb86ff436dc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 14:08:10 2021 +0200

    Add StatsD support to graph submodule
    
    Time stats of graphs creation and counter of amount of invalidated isochrone frontiers

commit a54cbad79133f9319acfe7eae4ab34fbd5ed887d
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 13:53:57 2021 +0200

    Add StatsD support to all three storage implementations

commit 4c5db2733f7695429f92b5335142ff7ea3803182
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:21:42 2021 +0200

    Add StatsD support to provenance backend

commit 2740c1611a364214acf7d6c292143043a9e827b1
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:17:34 2021 +0200

    Split `Provenance::flush` method in two (one per layer)

commit 8c0d6a735afcd07e102cd90de6e5c0a6531fb141
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.
    The new `ProvenanceStorageRabbitMQ` remote storage should be used instead.

commit 15fbf939c19db9f29c99b7c1d045f61967f3886c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 21 16:13:53 2021 +0200

    Add support for remote backend on existing storage tests

commit 2ca0c9bdf640c7cbbcf84580aa211d8cb51d55a4
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling

commit 0ec7250ea299d42697cb3b480171efcd2926d049
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit f0f3a584ea6965021990ddba926ae13c29b9560a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 846b20e0e9995a13591a1641bf92036ff3764be5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `open`/`close` methods to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly allocate/release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/402/ for more details.

Harbormaster completed remote builds in B24094: Diff 23178.Sep 29 2021, 2:47 PM
aeviso updated this revision to Diff 23186.Sep 29 2021, 3:38 PM

rebase

aeviso added a child revision: D6165: Add new RabbitMQ-based client/server API.Sep 29 2021, 3:41 PM
swh-public-ci added a comment.Sep 29 2021, 3:42 PM

Build is green

Patch application report for D6358 (id=23186)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..61b427c
Fast-forward
 requirements-test.txt                   |   1 -
 requirements.txt                        |   1 +
 swh/provenance/__init__.py              |  24 ++---
 swh/provenance/api/client.py            |  13 ---
 swh/provenance/api/server.py            |  92 +-------------------
 swh/provenance/cli.py                   |  13 ++-
 swh/provenance/graph.py                 |   9 ++
 swh/provenance/interface.py             |  20 +++++
 swh/provenance/mongo/backend.py         |  41 ++++++++-
 swh/provenance/origin.py                |  17 ++--
 swh/provenance/postgresql/archive.py    |  15 ++--
 swh/provenance/postgresql/provenance.py |  40 +++++++--
 swh/provenance/provenance.py            | 149 ++++++++++++++++++--------------
 swh/provenance/revision.py              |  31 +++----
 swh/provenance/storage/archive.py       |  15 ++--
 swh/provenance/tests/conftest.py        |  63 ++++++--------
 16 files changed, 253 insertions(+), 291 deletions(-)
Changes applied before test
commit 61b427c0956bd35596213df7c0f4655966227449
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:59:38 2021 +0200

    Make old StatsD metrics style compliant with the rest of the module

commit d9a00102c66284f358c6ced5e3fdf1057a9ba62d
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 14:08:10 2021 +0200

    Add StatsD support to graph submodule
    
    Time stats of graphs creation and counter of amount of invalidated isochrone frontiers

commit 0cf3d9185f3eb8528c9cf2031ea8f94d83977ca2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 13:53:57 2021 +0200

    Add StatsD support to provenance storage implementations

commit 0160d4f7c3cfc3f0193b729f0c04bd2ff7ad7129
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:21:42 2021 +0200

    Add StatsD support to provenance backend

commit 4f6bf0a4670e69730e47f519ac8bca6673be29f6
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:17:34 2021 +0200

    Split `Provenance::flush` method in two (one per layer)

commit 8d401db34539f5df2ce2bd37080ec8ae1557417b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.

commit 846b20e0e9995a13591a1641bf92036ff3764be5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `open`/`close` methods to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly allocate/release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/408/ for more details.

Harbormaster completed remote builds in B24103: Diff 23186.Sep 29 2021, 3:42 PM
douardda accepted this revision.Oct 1 2021, 1:59 PM

I still think it's best to use the wrapped function name as "method" but meh

This revision is now accepted and ready to land.Oct 1 2021, 1:59 PM
aeviso updated this revision to Diff 23273.Oct 4 2021, 3:53 PM

rebase

swh-public-ci added a comment.Oct 4 2021, 3:56 PM

Build is green

Patch application report for D6358 (id=23273)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..04ff73e
Fast-forward
 requirements-test.txt                   |   1 -
 requirements.txt                        |   2 +-
 swh/provenance/__init__.py              |  24 ++---
 swh/provenance/api/client.py            |  13 ---
 swh/provenance/api/server.py            |  92 +-----------------
 swh/provenance/cli.py                   |  71 +++++++-------
 swh/provenance/graph.py                 |   9 ++
 swh/provenance/interface.py             |  47 ++++++++-
 swh/provenance/mongo/backend.py         |  61 +++++++++++-
 swh/provenance/origin.py                |  17 +---
 swh/provenance/postgresql/archive.py    |  15 +--
 swh/provenance/postgresql/provenance.py |  58 +++++++++--
 swh/provenance/provenance.py            | 165 +++++++++++++++++++-------------
 swh/provenance/revision.py              |  31 +++---
 swh/provenance/storage/archive.py       |  15 +--
 swh/provenance/tests/conftest.py        |  61 +++++-------
 16 files changed, 349 insertions(+), 333 deletions(-)
Changes applied before test
commit 04ff73ea98f5f239cee6a126c75767f4617e330c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:59:38 2021 +0200

    Make old StatsD metrics style compliant with the rest of the module

commit 1bd6b22aae6a356e18f65005fc7e1c162e6f38c6
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 14:08:10 2021 +0200

    Add StatsD support to graph submodule
    
    Time stats of graphs creation and counter of amount of invalidated isochrone frontiers

commit 1ad78362fb415ea1d88a1d416da9991896e68d43
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 13:53:57 2021 +0200

    Add StatsD support to provenance storage implementations

commit e2a1843d5ebe01a9cdfe46b6b74dde1e293b8c01
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:21:42 2021 +0200

    Add StatsD support to provenance backend

commit 246e55f9b7e3475ea4509e08370827a3190db916
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:17:34 2021 +0200

    Split `Provenance::flush` method in two (one per layer)

commit f0210c3753c3a4122ee3c54f7fac97d170a142fa
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `open`/`close` methods to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    This allows to have an explicit mechanism to allocate/release resources when needed.
    The necessary methods for the classes implementing these interfaces to be turned in contexts
    managers are added as well (ie. `__enter__`/`__exit__`).

commit 172e327c25883bee768a9c16b850ce6aab7e2eb2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Remove remote provenance storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/424/ for more details.

Harbormaster completed remote builds in B24190: Diff 23273.Oct 4 2021, 3:56 PM
Closed by commit rDPROV04ff73ea98f5: Make old StatsD metrics style compliant with the rest of the module (authored by aeviso). · Explain WhyOct 5 2021, 12:05 PM
This revision was automatically updated to reflect the committed changes.
aeviso added a commit: rDPROV04ff73ea98f5: Make old StatsD metrics style compliant with the rest of the module.
aeviso removed a child revision: D6165: Add new RabbitMQ-based client/server API.Oct 5 2021, 2:39 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
provenance/
17 lines97%
Loading...
postgresql/
15 lines100%
Loading...
31 lines94%
Loading...
storage/
15 lines91%
Loading...

Diff 23178

View Options

swh/provenance/origin.py

Loading...
View Options

swh/provenance/postgresql/archive.py

Loading...
View Options

swh/provenance/revision.py

Loading...
View Options

swh/provenance/storage/archive.py

Loading...
About · Developer information · Legal