Page MenuHomeSoftware Heritage
Differential D6273

Remove remote storage based on `swh.core.api.RPCClient`
ClosedPublic
Actions

Authored by aeviso on Sep 15 2021, 4:27 PM.

Details

Reviewers
olasd
douardda
Group Reviewers
Reviewers
Commits
rDPROV172e327c2588: Remove remote provenance storage based on `swh.core.api.RPCClient`
Summary

Squashed with D6272

Diff Detail

Repository
rDPROV Provenance database
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

aeviso created this revision.Sep 15 2021, 4:27 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptSep 15 2021, 4:27 PM
aeviso added a reviewer: olasd.Sep 15 2021, 4:28 PM
swh-public-ci added a comment.Sep 15 2021, 4:30 PM

Build is green

Patch application report for D6273 (id=22710)

Could not rebase; Attempt merge onto 3383cae57e...

Updating 3383cae..17c6926
Fast-forward
 mypy.ini                                       |   3 +
 swh/provenance/__init__.py                     |  14 +-
 swh/provenance/api/client.py                   | 508 ++++++++++++++-
 swh/provenance/api/server.py                   | 818 ++++++++++++++++++++++---
 swh/provenance/cli.py                          |  29 +-
 swh/provenance/origin.py                       |  13 +
 swh/provenance/postgresql/archive.py           |  13 +
 swh/provenance/revision.py                     |  13 +
 swh/provenance/storage/archive.py              |  13 +
 swh/provenance/tests/conftest.py               |  31 +-
 swh/provenance/tests/test_archive_interface.py |  16 +-
 11 files changed, 1345 insertions(+), 126 deletions(-)
Changes applied before test
commit 17c692664f6da9593919c50460d8aec54819a242
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`

commit b11a446102843ed6197505ba18cfd6807b3f75aa
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic

commit 37e87534960abd9bcb5a5082ec933b4a542b3b21
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit d133b5448d26347ed761d678deb5f3ecd17ae86c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit e0509cb75b21e7594de55b66818b3d3d663e0dcc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes

commit d0a11a1499839baebf67407314a78029057b606a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 10:52:44 2021 +0200

    Add StatsD support to the main algorithms of the module

commit a8009be206dc1e414a69a708f826266b750b74cd
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 14 15:33:33 2021 +0200

    Add StatsD support to `ArchiveInterface` implementations

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/349/ for more details.

Harbormaster completed remote builds in B23643: Diff 22710.Sep 15 2021, 4:30 PM
aeviso requested review of this revision.Sep 15 2021, 4:30 PM
vlorentz added a subscriber: vlorentz.Sep 21 2021, 7:16 PM

Why?

vlorentz added a comment.Sep 21 2021, 7:26 PM

Looks like it is supposed to be replaced by the rabbitmq-based RPC? Please explain it in the commit/diff message.

aeviso added a comment.Sep 22 2021, 10:51 AM
In D6273#163625, @vlorentz wrote:

Why?

You mean, why are we removing this backend implentation? Because it doesn't serve our purposes, the one using RabbitMQ is the only remote implementation we have that guarantees conflict free writings to the storage.

aeviso updated this revision to Diff 23005.Sep 23 2021, 2:17 PM

rebase

swh-public-ci added a comment.Sep 23 2021, 2:20 PM

Build is green

Patch application report for D6273 (id=23005)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..3b7827e
Fast-forward
 mypy.ini                                |   3 +
 swh/provenance/__init__.py              |  14 +-
 swh/provenance/api/client.py            | 528 +++++++++++++++++++-
 swh/provenance/api/server.py            | 844 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  33 +-
 swh/provenance/interface.py             |   9 +
 swh/provenance/mongo/backend.py         |   3 +
 swh/provenance/postgresql/provenance.py |   3 +
 swh/provenance/provenance.py            |   3 +
 swh/provenance/tests/conftest.py        |  57 +--
 10 files changed, 1366 insertions(+), 131 deletions(-)
Changes applied before test
commit 3b7827e09876f3d355672455ac532ef7daf430e1
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.
    The new `ProvenanceStorageRabbitMQ` remote storage should be used instead.

commit 5b2d1183b3af02059731f6020186275c832281cc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    to explicitly release resources.

commit 728bccedf66d70c6d0f2a64efeec41b416039a57
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit 243061be5bcdd4bc1da9cc40e956b9186d924d84
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/362/ for more details.

Harbormaster completed remote builds in B23931: Diff 23005.Sep 23 2021, 2:20 PM
aeviso updated this revision to Diff 23009.Sep 23 2021, 3:40 PM

rebase

swh-public-ci added a comment.Sep 23 2021, 3:41 PM

Build has FAILED

Patch application report for D6273 (id=23009)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..b32f7ba
Fast-forward
 .gitignore                              |   3 +-
 mypy.ini                                |   3 +
 pytest.ini                              |   5 +
 requirements-test.txt                   |   1 +
 swh/provenance/__init__.py              |  14 +-
 swh/provenance/api/client.py            | 528 +++++++++++++++++++-
 swh/provenance/api/server.py            | 844 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  33 +-
 swh/provenance/interface.py             |   9 +
 swh/provenance/mongo/backend.py         |   3 +
 swh/provenance/postgresql/provenance.py |   3 +
 swh/provenance/provenance.py            |   3 +
 swh/provenance/tests/conftest.py        |  79 +--
 tox.ini                                 |   4 +-
 14 files changed, 1399 insertions(+), 133 deletions(-)
Changes applied before test
commit b32f7ba5c70d732c7c50992811716f976920f94d
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.
    The new `ProvenanceStorageRabbitMQ` remote storage should be used instead.

commit 365f297f9362cee1bbc78f0d1a178f0846a32444
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 21 16:13:53 2021 +0200

    Add support for remote backend on existing storage tests

commit 42a2ee4e90755bf97cbd52fc151933d495916d00
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    to explicitly release resources.

commit 728bccedf66d70c6d0f2a64efeec41b416039a57
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit 243061be5bcdd4bc1da9cc40e956b9186d924d84
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/364/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/364/console

Harbormaster failed remote builds in B23935: Diff 23009!Sep 23 2021, 3:41 PM
swh-public-ci added a comment.Sep 23 2021, 3:54 PM

Build has FAILED

Patch application report for D6273 (id=23009)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..b32f7ba
Fast-forward
 .gitignore                              |   3 +-
 mypy.ini                                |   3 +
 pytest.ini                              |   5 +
 requirements-test.txt                   |   1 +
 swh/provenance/__init__.py              |  14 +-
 swh/provenance/api/client.py            | 528 +++++++++++++++++++-
 swh/provenance/api/server.py            | 844 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  33 +-
 swh/provenance/interface.py             |   9 +
 swh/provenance/mongo/backend.py         |   3 +
 swh/provenance/postgresql/provenance.py |   3 +
 swh/provenance/provenance.py            |   3 +
 swh/provenance/tests/conftest.py        |  79 +--
 tox.ini                                 |   4 +-
 14 files changed, 1399 insertions(+), 133 deletions(-)
Changes applied before test
commit b32f7ba5c70d732c7c50992811716f976920f94d
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.
    The new `ProvenanceStorageRabbitMQ` remote storage should be used instead.

commit 365f297f9362cee1bbc78f0d1a178f0846a32444
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 21 16:13:53 2021 +0200

    Add support for remote backend on existing storage tests

commit 42a2ee4e90755bf97cbd52fc151933d495916d00
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    to explicitly release resources.

commit 728bccedf66d70c6d0f2a64efeec41b416039a57
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit 243061be5bcdd4bc1da9cc40e956b9186d924d84
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/365/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/365/console

aeviso updated this revision to Diff 23014.Sep 23 2021, 5:06 PM

rebase

swh-public-ci added a comment.Sep 23 2021, 5:13 PM

Build is green

Patch application report for D6273 (id=23014)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..f2ca78f
Fast-forward
 .gitignore                              |   3 +-
 mypy.ini                                |   3 +
 pytest.ini                              |   5 +
 requirements-test.txt                   |   1 +
 swh/provenance/__init__.py              |  14 +-
 swh/provenance/api/client.py            | 528 +++++++++++++++++++-
 swh/provenance/api/server.py            | 844 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  33 +-
 swh/provenance/interface.py             |   9 +
 swh/provenance/mongo/backend.py         |   3 +
 swh/provenance/postgresql/provenance.py |   3 +
 swh/provenance/provenance.py            |   3 +
 swh/provenance/tests/conftest.py        |  79 +--
 tox.ini                                 |   3 +-
 14 files changed, 1398 insertions(+), 133 deletions(-)
Changes applied before test
commit f2ca78f8f0b7e9240dfd623c2d22f29efbc7e707
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.
    The new `ProvenanceStorageRabbitMQ` remote storage should be used instead.

commit bf2c97ebd296eb9034279a36fbb9d8aaac4bfe5a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 21 16:13:53 2021 +0200

    Add support for remote backend on existing storage tests

commit 42a2ee4e90755bf97cbd52fc151933d495916d00
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    to explicitly release resources.

commit 728bccedf66d70c6d0f2a64efeec41b416039a57
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit 243061be5bcdd4bc1da9cc40e956b9186d924d84
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/368/ for more details.

Harbormaster completed remote builds in B23940: Diff 23014.Sep 23 2021, 5:13 PM
aeviso edited parent revisions, added: D6339: Add support for remote backend on existing storage tests; removed: D6165: Add new RabbitMQ-based client/server API.Sep 24 2021, 12:21 PM
aeviso updated this revision to Diff 23048.Sep 24 2021, 12:27 PM
aeviso added a parent revision: D6165: Add new RabbitMQ-based client/server API.

rebase

aeviso removed a parent revision: D6165: Add new RabbitMQ-based client/server API.Sep 24 2021, 12:27 PM
swh-public-ci added a comment.Sep 24 2021, 12:33 PM

Build is green

Patch application report for D6273 (id=23048)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..c936324
Fast-forward
 .gitignore                              |   4 +-
 mypy.ini                                |   3 +
 pytest.ini                              |   5 +
 requirements-test.txt                   |   1 +
 swh/provenance/__init__.py              |  14 +-
 swh/provenance/api/client.py            | 528 +++++++++++++++++++-
 swh/provenance/api/server.py            | 844 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  33 +-
 swh/provenance/interface.py             |   9 +
 swh/provenance/mongo/backend.py         |   3 +
 swh/provenance/postgresql/provenance.py |   3 +
 swh/provenance/provenance.py            |   3 +
 swh/provenance/tests/conftest.py        |  69 +--
 tox.ini                                 |   3 +-
 14 files changed, 1393 insertions(+), 129 deletions(-)
Changes applied before test
commit c93632431f97c67608a9130a2b01b7bfdbd2e753
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.
    The new `ProvenanceStorageRabbitMQ` remote storage should be used instead.

commit 50a1441f2068a2abe1293be893e66806cf0977b9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 21 16:13:53 2021 +0200

    Add support for remote backend on existing storage tests

commit ba22e90f707688aed716d20253256ee12c414a33
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling

commit 4ec6d3359e20920469cde43516bc120d4352f915
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit f6f174fb76b2c42435b6c75eb90c175d9cb0fca5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 437c2b4ad60e15c0b102988a99766b2c503c91d5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/375/ for more details.

Harbormaster completed remote builds in B23975: Diff 23048.Sep 24 2021, 12:33 PM
olasd added a comment.Sep 24 2021, 4:30 PM

I would suggest squashing D6272 and this together to land them at the same time.

I think you can remove types-werkzeug from requirements-test.txt. I'm not sure you can drop the http extra from swh.core dependencies in requirements-swh.txt, as the serialization/deserialization scaffolding is still in use in the rabbitmq backend.

aeviso added a comment.Sep 24 2021, 4:56 PM
In D6273#164555, @olasd wrote:

I would suggest squashing D6272 and this together to land them at the same time.

I think you can remove types-werkzeug from requirements-test.txt. I'm not sure you can drop the http extra from swh.core dependencies in requirements-swh.txt, as the serialization/deserialization scaffolding is still in use in the rabbitmq backend.

OK, about types-werkzeug. I'll remove it.

However, I would prefer not to squash this together with D6272 since moving those commits together in the git history is really a pain of conflict resolution (I even end up with empty files that later reappear).
There is no harm on keeping ProvenanceStorageRPC until ProvenanceStorageRabbitMQ is landed.

aeviso updated this revision to Diff 23062.Sep 24 2021, 4:57 PM
aeviso added a parent revision: D6165: Add new RabbitMQ-based client/server API.

rebase

swh-public-ci added a comment.Sep 24 2021, 5:02 PM

Build is green

Patch application report for D6273 (id=23062)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..4995134
Fast-forward
 .gitignore                              |   4 +-
 mypy.ini                                |   3 +
 pytest.ini                              |   5 +
 requirements-test.txt                   |   2 +-
 swh/provenance/__init__.py              |  14 +-
 swh/provenance/api/client.py            | 528 +++++++++++++++++++-
 swh/provenance/api/server.py            | 844 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  33 +-
 swh/provenance/interface.py             |   9 +
 swh/provenance/mongo/backend.py         |   3 +
 swh/provenance/postgresql/provenance.py |   3 +
 swh/provenance/provenance.py            |   3 +
 swh/provenance/tests/conftest.py        |  69 +--
 tox.ini                                 |   3 +-
 14 files changed, 1393 insertions(+), 130 deletions(-)
Changes applied before test
commit 4995134725cb5e62bcdf1d0c19ea4e2fabac5465
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.
    The new `ProvenanceStorageRabbitMQ` remote storage should be used instead.

commit 50a1441f2068a2abe1293be893e66806cf0977b9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 21 16:13:53 2021 +0200

    Add support for remote backend on existing storage tests

commit ba22e90f707688aed716d20253256ee12c414a33
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling

commit 4ec6d3359e20920469cde43516bc120d4352f915
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit f6f174fb76b2c42435b6c75eb90c175d9cb0fca5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 437c2b4ad60e15c0b102988a99766b2c503c91d5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/376/ for more details.

Harbormaster completed remote builds in B23989: Diff 23062.Sep 24 2021, 5:02 PM
olasd added a comment.Sep 24 2021, 5:37 PM
In D6273#164561, @aeviso wrote:

However, I would prefer not to squash this together with D6272 since moving those commits together in the git history is really a pain of conflict resolution (I even end up with empty files that later reappear).
There is no harm on keeping ProvenanceStorageRPC until ProvenanceStorageRabbitMQ is landed.

Oof, yeah, no problem.

aeviso added a child revision: D6351: Add StatsD support to provenance backend.Sep 27 2021, 2:11 PM
aeviso removed a parent revision: D6165: Add new RabbitMQ-based client/server API.Sep 27 2021, 2:20 PM
aeviso added a child revision: D6357: Split `Provenance::flush` method in two (one per layer).Sep 27 2021, 4:01 PM
aeviso removed a child revision: D6351: Add StatsD support to provenance backend.Sep 27 2021, 4:05 PM
aeviso updated this revision to Diff 23173.Sep 29 2021, 2:31 PM
aeviso added a parent revision: D6165: Add new RabbitMQ-based client/server API.

rebase

aeviso edited the summary of this revision. (Show Details)Sep 29 2021, 2:32 PM
aeviso removed a parent revision: D6165: Add new RabbitMQ-based client/server API.
swh-public-ci added a comment.Sep 29 2021, 2:39 PM

Build is green

Patch application report for D6273 (id=23173)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..8c0d6a7
Fast-forward
 .gitignore                              |   4 +-
 mypy.ini                                |   3 +
 pytest.ini                              |   5 +
 requirements-test.txt                   |   2 +-
 requirements.txt                        |   1 +
 swh/provenance/__init__.py              |  28 +-
 swh/provenance/api/client.py            | 530 +++++++++++++++++++-
 swh/provenance/api/server.py            | 844 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  39 +-
 swh/provenance/interface.py             |  20 +
 swh/provenance/mongo/backend.py         |  19 +-
 swh/provenance/postgresql/provenance.py |  18 +-
 swh/provenance/provenance.py            |   6 +
 swh/provenance/tests/conftest.py        |  84 ++--
 tox.ini                                 |   3 +-
 15 files changed, 1453 insertions(+), 153 deletions(-)
Changes applied before test
commit 8c0d6a735afcd07e102cd90de6e5c0a6531fb141
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.
    The new `ProvenanceStorageRabbitMQ` remote storage should be used instead.

commit 15fbf939c19db9f29c99b7c1d045f61967f3886c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 21 16:13:53 2021 +0200

    Add support for remote backend on existing storage tests

commit 2ca0c9bdf640c7cbbcf84580aa211d8cb51d55a4
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling

commit 0ec7250ea299d42697cb3b480171efcd2926d049
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit f0f3a584ea6965021990ddba926ae13c29b9560a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 846b20e0e9995a13591a1641bf92036ff3764be5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `open`/`close` methods to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly allocate/release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/397/ for more details.

Harbormaster completed remote builds in B24089: Diff 23173.Sep 29 2021, 2:39 PM
aeviso updated this revision to Diff 23180.Sep 29 2021, 3:31 PM

move earlier in the commit history

aeviso edited parent revisions, added: D6334: Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`; removed: D6339: Add support for remote backend on existing storage tests.Sep 29 2021, 3:32 PM
swh-public-ci added a comment.Sep 29 2021, 3:34 PM

Build is green

Patch application report for D6273 (id=23180)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..8d401db
Fast-forward
 requirements-test.txt                   |  1 -
 requirements.txt                        |  1 +
 swh/provenance/__init__.py              | 24 +++------
 swh/provenance/api/client.py            | 13 -----
 swh/provenance/api/server.py            | 92 +--------------------------------
 swh/provenance/cli.py                   | 13 +++--
 swh/provenance/interface.py             | 20 +++++++
 swh/provenance/mongo/backend.py         | 19 +++++--
 swh/provenance/postgresql/provenance.py | 18 ++++---
 swh/provenance/provenance.py            |  6 +++
 swh/provenance/tests/conftest.py        | 63 ++++++++++------------
 11 files changed, 97 insertions(+), 173 deletions(-)
Changes applied before test
commit 8d401db34539f5df2ce2bd37080ec8ae1557417b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.

commit 846b20e0e9995a13591a1641bf92036ff3764be5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `open`/`close` methods to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly allocate/release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/403/ for more details.

Harbormaster completed remote builds in B24096: Diff 23180.Sep 29 2021, 3:34 PM
douardda accepted this revision.Oct 1 2021, 2:30 PM
This revision is now accepted and ready to land.Oct 1 2021, 2:30 PM
aeviso updated this revision to Diff 23265.Oct 4 2021, 3:38 PM
aeviso mentioned this in D6272: Remove remote storage based on `swh.core.api.RPCClient`.

squash with D6272

aeviso edited the summary of this revision. (Show Details)Oct 4 2021, 3:41 PM
aeviso removed a parent revision: D6334: Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`.
swh-public-ci added a comment.Oct 4 2021, 3:42 PM

Build is green

Patch application report for D6273 (id=23265)

Rebasing onto 4c087ea0ec...

Current branch diff-target is up to date.
Changes applied before test
commit 68c54032460a8f9b97b48417404e910e88a84328
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Remove remote provenance storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/416/ for more details.

Harbormaster completed remote builds in B24182: Diff 23265.Oct 4 2021, 3:42 PM
aeviso retitled this revision from Remove old client/server storage based on `swh.core.api.RPCClient` to Remove remote storage based on `swh.core.api.RPCClient`.Oct 4 2021, 3:43 PM
aeviso updated this revision to Diff 23267.Oct 4 2021, 3:46 PM

rebase

swh-public-ci added a comment.Oct 4 2021, 3:49 PM

Build is green

Patch application report for D6273 (id=23267)

Rebasing onto 4c087ea0ec...

Current branch diff-target is up to date.
Changes applied before test
commit 172e327c25883bee768a9c16b850ce6aab7e2eb2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Remove remote provenance storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/418/ for more details.

Harbormaster completed remote builds in B24184: Diff 23267.Oct 4 2021, 3:49 PM
aeviso removed a child revision: D6357: Split `Provenance::flush` method in two (one per layer).Oct 4 2021, 3:50 PM
Closed by commit rDPROV172e327c2588: Remove remote provenance storage based on `swh.core.api.RPCClient` (authored by aeviso). · Explain WhyOct 5 2021, 12:05 PM
This revision was automatically updated to reflect the committed changes.
aeviso added a commit: rDPROV172e327c2588: Remove remote provenance storage based on `swh.core.api.RPCClient`.

Revision Contents
Changeset List

PathSize
1 line
1 line
swh/
provenance/
10 lines
api/
13 lines
92 lines
3 lines
tests/
31 lines

Diff 23304

View Options

requirements-test.txt

Loading...
View Options

requirements.txt

Loading...
View Options

swh/provenance/__init__.py

Loading...
View Options

swh/provenance/api/client.py

Loading...
View Options

swh/provenance/api/server.py

Loading...
View Options

swh/provenance/cli.py

Loading...
View Options

swh/provenance/tests/conftest.py

Loading...
About · Developer information · Legal