Page MenuHomeSoftware Heritage

Add new RabbitMQ-based client/server API
AcceptedPublic

Authored by aeviso on Aug 31 2021, 2:03 PM.

Details

Reviewers
olasd
douardda
Group Reviewers
Reviewers
Summary

New conflict resolution layer implementation using RabbitMQ to communicate
between client and server processes.

For each set methods in the ProvenanceStorageInterface the client will
dispatch the information to be stored to different queues, based on the id
of the associated entity (in case of a relation, the source entity).

The server will spawn one sub-process per queue to handle those particular
requirements. The split policy is defined in the server class in such a
way that no writing conflicts should occur in the underlying storage.

For the get methods, the client directly access the underlying storage
object, for which it has its own connection (ie. no communication through
RabbitMQ occurs).

Docs: https://hedgedoc.softwareheritage.org/RJQjBSR2TmuVzD6NRFFCeg

Diff Detail

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
  • Add new RabbitMQ-based client/server API
  • Rework ProvenanceStorageRabbitMQWorker to handle connection loss
  • Improve server/client shoutdown logic

Build is green

Patch application report for D6165 (id=22708)

Could not rebase; Attempt merge onto 3383cae57e...

Updating 3383cae..b11a446
Fast-forward
 mypy.ini                                       |   3 +
 swh/provenance/__init__.py                     |  29 +-
 swh/provenance/api/client.py                   | 517 ++++++++++++++++-
 swh/provenance/api/server.py                   | 766 ++++++++++++++++++++++++-
 swh/provenance/cli.py                          |  28 +-
 swh/provenance/origin.py                       |  13 +
 swh/provenance/postgresql/archive.py           |  13 +
 swh/provenance/revision.py                     |  13 +
 swh/provenance/storage/archive.py              |  13 +
 swh/provenance/tests/conftest.py               |  12 +-
 swh/provenance/tests/test_archive_interface.py |  16 +-
 11 files changed, 1391 insertions(+), 32 deletions(-)
Changes applied before test
commit b11a446102843ed6197505ba18cfd6807b3f75aa
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic

commit 37e87534960abd9bcb5a5082ec933b4a542b3b21
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit d133b5448d26347ed761d678deb5f3ecd17ae86c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit e0509cb75b21e7594de55b66818b3d3d663e0dcc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes

commit d0a11a1499839baebf67407314a78029057b606a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 10:52:44 2021 +0200

    Add StatsD support to the main algorithms of the module

commit a8009be206dc1e414a69a708f826266b750b74cd
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 14 15:33:33 2021 +0200

    Add StatsD support to `ArchiveInterface` implementations

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/348/ for more details.

aeviso retitled this revision from [WIP] Add new RabbitMQ-based client/server API to Add new RabbitMQ-based client/server API.Sep 15 2021, 4:26 PM

What is the reason for this change? Is it more efficient assign requests to workers based on ID rather than randomly?

What is the reason for this change? Is it more efficient assign requests to workers based on ID rather than randomly?

To go one step further, this Diff really needs some design documentation (with rationale, design choice and architecture).

What is the reason for this change? Is it more efficient assign requests to workers based on ID rather than randomly?

I don't understand the question. What do you mean by assigning requests randomly? How do you guarantee conflict resolution that way?

What do you mean by assigning requests randomly?

The way gunicorn assigns requests to worker processes in the existing RPC server

How do you guarantee conflict resolution that way?

What conflict resolution? Sorry I didn't really follow swh-provenance's development, and I can't find this in the documentation

What do you mean by assigning requests randomly?

The way gunicorn assigns requests to worker processes in the existing RPC server

How do you guarantee conflict resolution that way?

What conflict resolution? Sorry I didn't really follow swh-provenance's development, and I can't find this in the documentation

It's actually explained in the document:

There are two possible kinds of conflicts:

same entity inserted by two distinct workers: it should be resolved by keeping the earliest date.
same relation inserted by two distinct workers: one of them can be safely ignored.

The whole purpose of this layer is to guarantee that we can insert in the database without write conflicts, while avoiding the use of locks. It's all described in the documentation.
I don't see how randomly assigning the workers can guarantee that. It might reduce the chance of a conflict, but not avoid it completely (IMO).

It's actually explained in the document:

Oh, sorry, I missed the hedgedoc link. I only looked in the repo and the diff's content.

Could you document the new design in this diff too, in the docs/ folder?

There are two possible kinds of conflicts:

same entity inserted by two distinct workers: it should be resolved by keeping the earliest date.
same relation inserted by two distinct workers: one of them can be safely ignored.

The whole purpose of this layer is to guarantee that we can insert in the database without write conflicts, while avoiding the use of locks. It's all described in the documentation.
I don't see how randomly assigning the workers can guarantee that. It might reduce the chance of a conflict, but not avoid it completely (IMO).

Ah, I see. I didn't know we had both readers and writers.

It's actually explained in the document:

Oh, sorry, I missed the hedgedoc link. I only looked in the repo and the diff's content.

Could you document the new design in this diff too, in the docs/ folder?

Sure. I've never used the docs before though. Can you point me to an example of what's expected to be included in that directory?

It could be something like these:

You can start from your Hedgedoc document, remove the description of the current state, but keep the description of the new design and the rationale

It could be something like these:

You can start from your Hedgedoc document, remove the description of the current state, but keep the description of the new design and the rationale

Thanks, but that's not what I meant. I mean what file format are you expecting to find in the docs folder? A HTML one?

Build is green

Patch application report for D6165 (id=23004)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..5b2d118
Fast-forward
 mypy.ini                                |   3 +
 swh/provenance/__init__.py              |  29 +-
 swh/provenance/api/client.py            | 537 +++++++++++++++++++++-
 swh/provenance/api/server.py            | 792 +++++++++++++++++++++++++++++++-
 swh/provenance/cli.py                   |  34 +-
 swh/provenance/interface.py             |   9 +
 swh/provenance/mongo/backend.py         |   3 +
 swh/provenance/postgresql/provenance.py |   3 +
 swh/provenance/provenance.py            |   3 +
 swh/provenance/tests/conftest.py        |  46 +-
 10 files changed, 1417 insertions(+), 42 deletions(-)
Changes applied before test
commit 5b2d1183b3af02059731f6020186275c832281cc
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    to explicitly release resources.

commit 728bccedf66d70c6d0f2a64efeec41b416039a57
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit 243061be5bcdd4bc1da9cc40e956b9186d924d84
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/361/ for more details.

It should be reStructuredText (introduction here: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html ), with a reference from the toctree in docs/index.rst

  • Add support for remote backend on existing storage tests

Build has FAILED

Patch application report for D6165 (id=23008)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..365f297
Fast-forward
 .gitignore                              |   3 +-
 mypy.ini                                |   3 +
 pytest.ini                              |   5 +
 requirements-test.txt                   |   1 +
 swh/provenance/__init__.py              |  29 +-
 swh/provenance/api/client.py            | 537 +++++++++++++++++++++-
 swh/provenance/api/server.py            | 792 +++++++++++++++++++++++++++++++-
 swh/provenance/cli.py                   |  34 +-
 swh/provenance/interface.py             |   9 +
 swh/provenance/mongo/backend.py         |   3 +
 swh/provenance/postgresql/provenance.py |   3 +
 swh/provenance/provenance.py            |   3 +
 swh/provenance/tests/conftest.py        |  69 ++-
 tox.ini                                 |   4 +-
 14 files changed, 1451 insertions(+), 44 deletions(-)
Changes applied before test
commit 365f297f9362cee1bbc78f0d1a178f0846a32444
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 21 16:13:53 2021 +0200

    Add support for remote backend on existing storage tests

commit 42a2ee4e90755bf97cbd52fc151933d495916d00
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    to explicitly release resources.

commit 728bccedf66d70c6d0f2a64efeec41b416039a57
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit 243061be5bcdd4bc1da9cc40e956b9186d924d84
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

Link to build: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/363/
See console output for more information: https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/363/console

Build is green

Patch application report for D6165 (id=23013)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..bf2c97e
Fast-forward
 .gitignore                              |   3 +-
 mypy.ini                                |   3 +
 pytest.ini                              |   5 +
 requirements-test.txt                   |   1 +
 swh/provenance/__init__.py              |  29 +-
 swh/provenance/api/client.py            | 537 +++++++++++++++++++++-
 swh/provenance/api/server.py            | 792 +++++++++++++++++++++++++++++++-
 swh/provenance/cli.py                   |  34 +-
 swh/provenance/interface.py             |   9 +
 swh/provenance/mongo/backend.py         |   3 +
 swh/provenance/postgresql/provenance.py |   3 +
 swh/provenance/provenance.py            |   3 +
 swh/provenance/tests/conftest.py        |  69 ++-
 tox.ini                                 |   3 +-
 14 files changed, 1450 insertions(+), 44 deletions(-)
Changes applied before test
commit bf2c97ebd296eb9034279a36fbb9d8aaac4bfe5a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Sep 21 16:13:53 2021 +0200

    Add support for remote backend on existing storage tests

commit 42a2ee4e90755bf97cbd52fc151933d495916d00
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    to explicitly release resources.

commit 728bccedf66d70c6d0f2a64efeec41b416039a57
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit 243061be5bcdd4bc1da9cc40e956b9186d924d84
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/367/ for more details.

  • Add new RabbitMQ-based client/server API
  • Rework ProvenanceStorageRabbitMQWorker to handle connection loss
  • Improve server/client shoutdown logic and error handling

Build is green

Patch application report for D6165 (id=23044)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..2e43218
Fast-forward
 mypy.ini                                |   3 +
 swh/provenance/__init__.py              |  29 +-
 swh/provenance/api/client.py            | 537 +++++++++++++++++++++-
 swh/provenance/api/server.py            | 792 +++++++++++++++++++++++++++++++-
 swh/provenance/cli.py                   |  34 +-
 swh/provenance/interface.py             |   9 +
 swh/provenance/mongo/backend.py         |   3 +
 swh/provenance/postgresql/provenance.py |   3 +
 swh/provenance/provenance.py            |   3 +
 swh/provenance/tests/conftest.py        |  35 +-
 10 files changed, 1411 insertions(+), 37 deletions(-)
Changes applied before test
commit 2e432185109a272a03aad42d7d380bd6ae2de4be
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling

commit 4ec6d3359e20920469cde43516bc120d4352f915
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit f6f174fb76b2c42435b6c75eb90c175d9cb0fca5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 437c2b4ad60e15c0b102988a99766b2c503c91d5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/371/ for more details.

Build is green

Patch application report for D6165 (id=23046)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..ba22e90
Fast-forward
 mypy.ini                                |   3 +
 swh/provenance/__init__.py              |  29 +-
 swh/provenance/api/client.py            | 537 +++++++++++++++++++++-
 swh/provenance/api/server.py            | 792 +++++++++++++++++++++++++++++++-
 swh/provenance/cli.py                   |  34 +-
 swh/provenance/interface.py             |   9 +
 swh/provenance/mongo/backend.py         |   3 +
 swh/provenance/postgresql/provenance.py |   3 +
 swh/provenance/provenance.py            |   3 +
 swh/provenance/tests/conftest.py        |  33 +-
 10 files changed, 1410 insertions(+), 36 deletions(-)
Changes applied before test
commit ba22e90f707688aed716d20253256ee12c414a33
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling

commit 4ec6d3359e20920469cde43516bc120d4352f915
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit f6f174fb76b2c42435b6c75eb90c175d9cb0fca5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 437c2b4ad60e15c0b102988a99766b2c503c91d5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `close` method to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/373/ for more details.

Thanks for this massive implementation work!

I still want to do a deeper dive in this code (and give others the chance to do so), but I think that before that, and now that bugs and wrinkles have been ironed out and this code seems to be working, we need a large pass of updating the docstrings to describe the actual behavior of the code.

I expect a lot of this is present inside the hedgedoc document, so you should try to land it as documentation at the same time as this code.

When reading this diff, I would like to find the following:

  • a description of all threads and subprocesses (on the client and server side), as well as their associated workflows (who does what)
  • a description of how RabbitMQ queues and exchanges are handled (the request queues, the response queues, the way the acknowledgements are managed)
  • a description of how objects are serialised to be passed on to the queues
  • a description of what queues feed to what server processes, and how the messages are "bundled" before being sent to the database
  • a list of "tunables" (number of queues, batch sizes, timeouts, etc.) to watch out for

I would suggest documenting the "lifecycle" of the client and server threads/processes, for instance by writing a summarised list of all the methods that are called in sequence, on initialization of the classes, with how the callbacks mesh together.

When this lifecycle doc is available (centrally), I think most of the "boilerplate" documentation that's been pulled from the pika example code can go away (with a shorter reference to the full lifecycle documentation).

In D6165#164547, @olasd wrote:

Thanks for this massive implementation work!

I still want to do a deeper dive in this code (and give others the chance to do so), but I think that before that, and now that bugs and wrinkles have been ironed out and this code seems to be working, we need a large pass of updating the docstrings to describe the actual behavior of the code.

I expect a lot of this is present inside the hedgedoc document, so you should try to land it as documentation at the same time as this code.

When reading this diff, I would like to find the following:

  • a description of all threads and subprocesses (on the client and server side), as well as their associated workflows (who does what)
  • a description of how RabbitMQ queues and exchanges are handled (the request queues, the response queues, the way the acknowledgements are managed)
  • a description of how objects are serialised to be passed on to the queues
  • a description of what queues feed to what server processes, and how the messages are "bundled" before being sent to the database
  • a list of "tunables" (number of queues, batch sizes, timeouts, etc.) to watch out for

I would suggest documenting the "lifecycle" of the client and server threads/processes, for instance by writing a summarised list of all the methods that are called in sequence, on initialization of the classes, with how the callbacks mesh together.

When this lifecycle doc is available (centrally), I think most of the "boilerplate" documentation that's been pulled from the pika example code can go away (with a shorter reference to the full lifecycle documentation).

I agree about documentation. My only concern is that this will delay the experiments on the server (and other machines if possible), and I'll actually have plenty of time to work on docs while those experiments run.

Build is green

Patch application report for D6165 (id=23170)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..2ca0c9b
Fast-forward
 mypy.ini                                |   3 +
 requirements.txt                        |   1 +
 swh/provenance/__init__.py              |  43 +-
 swh/provenance/api/client.py            | 537 ++++++++++++++++++++-
 swh/provenance/api/server.py            | 793 +++++++++++++++++++++++++++++++-
 swh/provenance/cli.py                   |  40 +-
 swh/provenance/interface.py             |  20 +
 swh/provenance/mongo/backend.py         |  19 +-
 swh/provenance/postgresql/provenance.py |  18 +-
 swh/provenance/provenance.py            |   6 +
 swh/provenance/tests/conftest.py        |  52 ++-
 11 files changed, 1473 insertions(+), 59 deletions(-)
Changes applied before test
commit 2ca0c9bdf640c7cbbcf84580aa211d8cb51d55a4
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling

commit 0ec7250ea299d42697cb3b480171efcd2926d049
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit f0f3a584ea6965021990ddba926ae13c29b9560a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 846b20e0e9995a13591a1641bf92036ff3764be5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `open`/`close` methods to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly allocate/release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/395/ for more details.

move later in the commit history

Build is green

Patch application report for D6165 (id=23187)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..efdce8a
Fast-forward
 mypy.ini                                |   3 +
 requirements-test.txt                   |   1 -
 requirements.txt                        |   1 +
 swh/provenance/__init__.py              |  28 +-
 swh/provenance/api/client.py            | 545 ++++++++++++++++++++-
 swh/provenance/api/server.py            | 844 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  39 +-
 swh/provenance/graph.py                 |   9 +
 swh/provenance/interface.py             |  20 +
 swh/provenance/mongo/backend.py         |  41 +-
 swh/provenance/origin.py                |  17 +-
 swh/provenance/postgresql/archive.py    |  15 +-
 swh/provenance/postgresql/provenance.py |  40 +-
 swh/provenance/provenance.py            | 149 +++---
 swh/provenance/revision.py              |  31 +-
 swh/provenance/storage/archive.py       |  15 +-
 swh/provenance/tests/conftest.py        |  63 +--
 17 files changed, 1592 insertions(+), 269 deletions(-)
Changes applied before test
commit efdce8a40cfd5864efbe6971dd26365d133b9260
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add StatsD support to client to be compliant with the other provenance
    storage implementations

commit 2049a13dcd69b2e9f5d853f75158967ecaf0ead3
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit 462d9f84facf6d7020000673d57000326df9dd4a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 61b427c0956bd35596213df7c0f4655966227449
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:59:38 2021 +0200

    Make old StatsD metrics style compliant with the rest of the module

commit d9a00102c66284f358c6ced5e3fdf1057a9ba62d
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 14:08:10 2021 +0200

    Add StatsD support to graph submodule
    
    Time stats of graphs creation and counter of amount of invalidated isochrone frontiers

commit 0cf3d9185f3eb8528c9cf2031ea8f94d83977ca2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 13:53:57 2021 +0200

    Add StatsD support to provenance storage implementations

commit 0160d4f7c3cfc3f0193b729f0c04bd2ff7ad7129
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:21:42 2021 +0200

    Add StatsD support to provenance backend

commit 4f6bf0a4670e69730e47f519ac8bca6673be29f6
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:17:34 2021 +0200

    Split `Provenance::flush` method in two (one per layer)

commit 8d401db34539f5df2ce2bd37080ec8ae1557417b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.

commit 846b20e0e9995a13591a1641bf92036ff3764be5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `open`/`close` methods to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly allocate/release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/409/ for more details.

Build is green

Patch application report for D6165 (id=23193)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..ca1ac92
Fast-forward
 mypy.ini                                |   3 +
 requirements-test.txt                   |   1 -
 requirements.txt                        |   1 +
 swh/provenance/__init__.py              |  28 +-
 swh/provenance/api/client.py            | 541 +++++++++++++++++++-
 swh/provenance/api/server.py            | 848 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  39 +-
 swh/provenance/graph.py                 |   9 +
 swh/provenance/interface.py             |  20 +
 swh/provenance/mongo/backend.py         |  41 +-
 swh/provenance/origin.py                |  17 +-
 swh/provenance/postgresql/archive.py    |  15 +-
 swh/provenance/postgresql/provenance.py |  40 +-
 swh/provenance/provenance.py            | 149 +++---
 swh/provenance/revision.py              |  31 +-
 swh/provenance/storage/archive.py       |  15 +-
 swh/provenance/tests/conftest.py        |  63 +--
 17 files changed, 1590 insertions(+), 271 deletions(-)
Changes applied before test
commit ca1ac92d5bddc8abad4a6b2283373ccf88acc3f3
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add StatsD support to client to be compliant with the other provenance
    storage implementations

commit ddd49fbd1841475f30faeb9257839a936cf2a815
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit a74ed65b748dd6bf3d1ec7f694a80889a7bb3b35
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 61b427c0956bd35596213df7c0f4655966227449
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:59:38 2021 +0200

    Make old StatsD metrics style compliant with the rest of the module

commit d9a00102c66284f358c6ced5e3fdf1057a9ba62d
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 14:08:10 2021 +0200

    Add StatsD support to graph submodule
    
    Time stats of graphs creation and counter of amount of invalidated isochrone frontiers

commit 0cf3d9185f3eb8528c9cf2031ea8f94d83977ca2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 13:53:57 2021 +0200

    Add StatsD support to provenance storage implementations

commit 0160d4f7c3cfc3f0193b729f0c04bd2ff7ad7129
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:21:42 2021 +0200

    Add StatsD support to provenance backend

commit 4f6bf0a4670e69730e47f519ac8bca6673be29f6
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:17:34 2021 +0200

    Split `Provenance::flush` method in two (one per layer)

commit 8d401db34539f5df2ce2bd37080ec8ae1557417b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.

commit 846b20e0e9995a13591a1641bf92036ff3764be5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `open`/`close` methods to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly allocate/release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/411/ for more details.

Build is green

Patch application report for D6165 (id=23213)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..5992c08
Fast-forward
 mypy.ini                                |   3 +
 requirements-test.txt                   |   1 -
 requirements.txt                        |   2 +
 swh/provenance/__init__.py              |  28 +-
 swh/provenance/api/client.py            | 541 +++++++++++++++++++-
 swh/provenance/api/server.py            | 848 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  39 +-
 swh/provenance/graph.py                 |   9 +
 swh/provenance/interface.py             |  20 +
 swh/provenance/mongo/backend.py         |  41 +-
 swh/provenance/origin.py                |  17 +-
 swh/provenance/postgresql/archive.py    |  15 +-
 swh/provenance/postgresql/provenance.py |  40 +-
 swh/provenance/provenance.py            | 149 +++---
 swh/provenance/revision.py              |  31 +-
 swh/provenance/storage/archive.py       |  15 +-
 swh/provenance/tests/conftest.py        |  63 +--
 17 files changed, 1591 insertions(+), 271 deletions(-)
Changes applied before test
commit 5992c08ff5c8f4eb2cdd8c5148375a050a4da738
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add StatsD support to client to be compliant with the other provenance
    storage implementations

commit 7513e6eff6979f8014dc23ebe0ea5c9e937de53c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit af1fe50f5a6548c6f4c31adbcf8d5124796d691b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 61b427c0956bd35596213df7c0f4655966227449
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:59:38 2021 +0200

    Make old StatsD metrics style compliant with the rest of the module

commit d9a00102c66284f358c6ced5e3fdf1057a9ba62d
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 14:08:10 2021 +0200

    Add StatsD support to graph submodule
    
    Time stats of graphs creation and counter of amount of invalidated isochrone frontiers

commit 0cf3d9185f3eb8528c9cf2031ea8f94d83977ca2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 13:53:57 2021 +0200

    Add StatsD support to provenance storage implementations

commit 0160d4f7c3cfc3f0193b729f0c04bd2ff7ad7129
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:21:42 2021 +0200

    Add StatsD support to provenance backend

commit 4f6bf0a4670e69730e47f519ac8bca6673be29f6
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:17:34 2021 +0200

    Split `Provenance::flush` method in two (one per layer)

commit 8d401db34539f5df2ce2bd37080ec8ae1557417b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 1 11:27:02 2021 +0200

    Remove old client/server storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.

commit 846b20e0e9995a13591a1641bf92036ff3764be5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `open`/`close` methods to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    The idea is to have a mechanism to explicitly allocate/release resources when needed.

commit 6c3071493b5d3f187113493275d402a27866da95
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Rename remote storage backend classes
    
    Make names consistent with the naming convention used for other components.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/413/ for more details.

Build is green

Patch application report for D6165 (id=23275)

Could not rebase; Attempt merge onto 4c087ea0ec...

Updating 4c087ea..cd4056b
Fast-forward
 mypy.ini                                |   3 +
 requirements-test.txt                   |   1 -
 requirements.txt                        |   3 +-
 swh/provenance/__init__.py              |  28 +-
 swh/provenance/api/client.py            | 557 ++++++++++++++++++++-
 swh/provenance/api/server.py            | 846 +++++++++++++++++++++++++++++---
 swh/provenance/cli.py                   |  97 ++--
 swh/provenance/graph.py                 |   9 +
 swh/provenance/interface.py             |  47 +-
 swh/provenance/mongo/backend.py         |  61 ++-
 swh/provenance/origin.py                |  17 +-
 swh/provenance/postgresql/archive.py    |  15 +-
 swh/provenance/postgresql/provenance.py |  58 ++-
 swh/provenance/provenance.py            | 165 ++++---
 swh/provenance/revision.py              |  31 +-
 swh/provenance/storage/archive.py       |  15 +-
 swh/provenance/tests/conftest.py        |  61 +--
 17 files changed, 1701 insertions(+), 313 deletions(-)
Changes applied before test
commit cd4056be39e8152276cc5d65f39d512021714d84
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add StatsD support to client to be compliant with the other provenance
    storage implementations

commit 2eaf7200e8a97e42b291313f04cf24ae6c6ce9f2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit d1913496c81be2fae1eff1cf93a17c8439991706
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 04ff73ea98f5f239cee6a126c75767f4617e330c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:59:38 2021 +0200

    Make old StatsD metrics style compliant with the rest of the module

commit 1bd6b22aae6a356e18f65005fc7e1c162e6f38c6
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 14:08:10 2021 +0200

    Add StatsD support to graph submodule
    
    Time stats of graphs creation and counter of amount of invalidated isochrone frontiers

commit 1ad78362fb415ea1d88a1d416da9991896e68d43
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 13:53:57 2021 +0200

    Add StatsD support to provenance storage implementations

commit e2a1843d5ebe01a9cdfe46b6b74dde1e293b8c01
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:21:42 2021 +0200

    Add StatsD support to provenance backend

commit 246e55f9b7e3475ea4509e08370827a3190db916
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Sep 27 15:17:34 2021 +0200

    Split `Provenance::flush` method in two (one per layer)

commit f0210c3753c3a4122ee3c54f7fac97d170a142fa
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Sep 24 11:08:08 2021 +0200

    Add `open`/`close` methods to both `ProvenanceInterface` and `ProvenanceStorageInterface`
    
    This allows to have an explicit mechanism to allocate/release resources when needed.
    The necessary methods for the classes implementing these interfaces to be turned in contexts
    managers are added as well (ie. `__enter__`/`__exit__`).

commit 172e327c25883bee768a9c16b850ce6aab7e2eb2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 16:14:10 2021 +0200

    Remove remote provenance storage based on `swh.core.api.RPCClient`
    
    This implementation was a first attempt for conflict resolution that didn't worked as expected.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/425/ for more details.

As others (and I) said, this must come with actual documentation.
As is, I have hard time understanding how this actually works (even after reading the document in hedgdoc).

Before landing this, please give credit to the origin of the example code used as starting point of a substantial part of the code in this diff (and make sure there is not license caveat).

This revision is now accepted and ready to land.Tue, Oct 5, 10:39 AM

Also there is no real value in keeping 3 revisions: the last 2 revisions actually improve/modify the code from the first revision.

  • Add new RabbitMQ-based client/server API
  • Rework ProvenanceStorageRabbitMQWorker to handle connection loss
  • Improve server/client shoutdown logic and error handling
  • Improve routing key computation for paths
  • Fix config file parsing for server initilization
  • Send several items per message in the remote provenance storage

Build is green

Patch application report for D6165 (id=23453)

Could not rebase; Attempt merge onto 04ff73ea98...

Updating 04ff73e..3f56270
Fast-forward
 mypy.ini                                        |   3 +
 requirements.txt                                |   1 +
 swh/provenance/__init__.py                      |   8 +
 swh/provenance/api/client.py                    | 582 +++++++++++++++++
 swh/provenance/api/server.py                    | 794 +++++++++++++++++++++++-
 swh/provenance/cli.py                           |  26 +-
 swh/provenance/model.py                         |   6 +-
 swh/provenance/provenance.py                    |  11 +-
 swh/provenance/tests/test_provenance_storage.py |  21 +-
 swh/provenance/util.py                          |  15 +
 10 files changed, 1432 insertions(+), 35 deletions(-)
 create mode 100644 swh/provenance/util.py
Changes applied before test
commit 3f56270a0f912e909312e52f864081bf6720cfce
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Oct 11 16:06:03 2021 +0200

    Send several items per message in the remote provenance storage

commit 03dc27f6f2eb1d99084fbc3a3f9ecdaa7c9edb27
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Oct 8 14:49:44 2021 +0200

    Fix config file parsing for server initilization

commit 2759b4977f933be21951ea90fd70f7c16c69aea1
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Oct 8 14:41:42 2021 +0200

    Improve routing key computation for paths

commit f146fac61a1ac44f489739caba3fe2b2f21de8d3
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add StatsD support to client to be compliant with the other provenance
    storage implementations

commit 327be11571eef3e44c38705990f9c931661a7591
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit bccbf59fcb16d8727d347c3d7b9a623704a80467
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 3e87301a2868a7a9aa42403e150a60489f22708e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Oct 8 14:40:39 2021 +0200

    Move path normalization function to `util` submodule

commit 2c9ef5673b369f2baa83b11ec9256c6aafc3a855
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Oct 5 12:01:25 2021 +0200

    Remove direct dependencies on deprecated `swh.model.identifiers` module

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/436/ for more details.

Build is green

Patch application report for D6165 (id=23517)

Could not rebase; Attempt merge onto 3e87301a28...

Updating 3e87301..b5a1d84
Fast-forward
 mypy.ini                         |   3 +
 requirements.txt                 |   1 +
 swh/provenance/__init__.py       |   8 +
 swh/provenance/api/client.py     | 582 ++++++++++++++++++++++++++++
 swh/provenance/api/server.py     | 794 ++++++++++++++++++++++++++++++++++++++-
 swh/provenance/cli.py            |  26 +-
 swh/provenance/sql/30-schema.sql |  20 +-
 swh/provenance/sql/40-funcs.sql  |  50 +--
 swh/provenance/util.py           |   5 +
 9 files changed, 1446 insertions(+), 43 deletions(-)
Changes applied before test
commit b5a1d8414f313af6a59211afa66816228c14172c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Oct 11 16:06:03 2021 +0200

    Send several items per message in the remote provenance storage

commit 5eb1fc59d3d34890f66f43314217412310fd919a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Oct 8 14:49:44 2021 +0200

    Fix config file parsing for server initilization

commit ce649ddb6db25dc3e68f128a7ae6174b8b31e8a0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Oct 8 14:41:42 2021 +0200

    Improve routing key computation for paths

commit 87108b1ea2ceb9909d4bbec6012004acdf96c08a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add StatsD support to client to be compliant with the other provenance
    storage implementations

commit e6588cb24368b5edbc8bfdb82b5a4ba9b690d444
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit 5a85b6c17f172315e784724158b7e08b6bdf9c61
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 37da3774d8dc34365b7b1cbed469d970c51ecc58
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Oct 14 12:03:47 2021 +0200

    Improve PostgreSQL storage scheme for the `with-path-denormalized` flavor
    
    Previous version was storing arrays of strings representing tuples for the
    denormalized relations (`dst` and `loc` of the relation resp.). While that
    simplified the check for duplicates, it turned out to be very inefficient
    in terms of disk usage. The new version has two distinct lists if `bigint`
    (ie. internal ids) for `dst` and `loc` resp. To check for duplicates the
    lists should be zipped, and repeated tuples filtered.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/444/ for more details.

  • Fix config file parsing for server initilization
  • Send several items per message in the remote provenance storage
  • Export batch size and prefetch count as parameters for remote storage

Build is green

Patch application report for D6165 (id=23635)

Could not rebase; Attempt merge onto 3e87301a28...

Updating 3e87301..effb5b0
Fast-forward
 mypy.ini                                |   3 +
 requirements.txt                        |   1 +
 swh/provenance/__init__.py              |   8 +
 swh/provenance/api/client.py            | 588 +++++++++++++++++++++++
 swh/provenance/api/server.py            | 808 +++++++++++++++++++++++++++++++-
 swh/provenance/cli.py                   |  31 +-
 swh/provenance/graph.py                 |   2 +-
 swh/provenance/postgresql/provenance.py |  29 +-
 swh/provenance/provenance.py            |  63 +++
 swh/provenance/sql/30-schema.sql        |  20 +-
 swh/provenance/sql/40-funcs.sql         |  50 +-
 swh/provenance/util.py                  |   5 +
 12 files changed, 1557 insertions(+), 51 deletions(-)
Changes applied before test
commit effb5b099a9e6928da42cdb491db532d6a75e988
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Oct 18 11:52:04 2021 +0200

    Export batch size and prefetch count as parameters for remote storage

commit 6e11a8e528850ae05375243469ed74ab9e0956ee
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Oct 11 16:06:03 2021 +0200

    Send several items per message in the remote provenance storage

commit d18e8bd4e9ca7814c9cdbfa7a21c155a3d7d3a08
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Oct 8 14:49:44 2021 +0200

    Fix config file parsing for server initilization

commit 018ab5106e5ac9be36830c8242f1d94c229b878a
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Oct 8 14:41:42 2021 +0200

    Improve routing key computation for paths

commit 647d0ae75b85043b0c2ef0f528be9f6891c91ce9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add StatsD support to client to be compliant with the other provenance
    storage implementations

commit cdddb2d573763ab005c7d3c754c5d85a263220e9
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit 35f03480581d52d1b7b705d0b974151fa49ba546
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 8168ab4fc3f0fc3556623dd3de854f222ffe5d7e
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Oct 14 12:03:47 2021 +0200

    Improve PostgreSQL storage scheme for the `with-path-denormalized` flavor
    
    Previous version was storing arrays of strings representing tuples for the
    denormalized relations (`dst` and `loc` of the relation resp.). While that
    simplified the check for duplicates, it turned out to be very inefficient
    in terms of disk usage. The new version has two distinct lists if `bigint`
    (ie. internal ids) for `dst` and `loc` resp. To check for duplicates the
    lists should be zipped, and repeated tuples filtered.

commit c7ae90e08b39919da9d67ad3436a71d47a6ad5e7
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Oct 18 12:10:10 2021 +0200

    Add metrics on retries when flushing cache on the provenance backend

commit bfea53a97c588aa85ddd2ea93fa3dcf17b34a6a4
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Oct 19 16:12:23 2021 +0200

    Export page size as a parameter for postgresql storage

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/459/ for more details.

Build is green

Patch application report for D6165 (id=23660)

Could not rebase; Attempt merge onto ef49e3100c...

Updating ef49e31..b280343
Fast-forward
 mypy.ini                         |   3 +
 requirements.txt                 |   1 +
 swh/provenance/__init__.py       |   8 +
 swh/provenance/api/client.py     | 588 ++++++++++++++++++++++++++++
 swh/provenance/api/server.py     | 808 ++++++++++++++++++++++++++++++++++++++-
 swh/provenance/cli.py            |  31 +-
 swh/provenance/sql/30-schema.sql |  20 +-
 swh/provenance/sql/40-funcs.sql  |  50 ++-
 swh/provenance/util.py           |   5 +
 9 files changed, 1470 insertions(+), 44 deletions(-)
Changes applied before test
commit b2803436de6bc67c6d0dc01b5624ebba18689ca2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Oct 18 11:52:04 2021 +0200

    Export batch size and prefetch count as parameters for remote storage

commit 3fee8fbfbdaac3ed1adb5003adb32c52c99c8d37
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Mon Oct 11 16:06:03 2021 +0200

    Send several items per message in the remote provenance storage

commit d8c0aae0093f700f5e362d60fe8cf6b51f374fa2
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Oct 8 14:49:44 2021 +0200

    Fix config file parsing for server initilization

commit 8e0d98fb67ace71342867b68c0703b551e47e7f0
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Oct 8 14:41:42 2021 +0200

    Improve routing key computation for paths

commit 1e66d6940749e33ff4442dfe8c2495567b6c50a5
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Wed Sep 15 13:39:59 2021 +0200

    Improve server/client shoutdown logic and error handling
    
    Add StatsD support to client to be compliant with the other provenance
    storage implementations

commit 5e8c8a54cdac94a2a41662468f33893df97e7c6b
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Tue Aug 31 13:36:34 2021 +0200

    Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
    
    Use `pika.SelectConnection` and make an explicit handle of its life-cycle.
    Improve connection error handling on both client and server side.
    
    Change the RabbitMQ scheme to use 5 exchanges (one per entity + location).
    Each exchange handles all entity related insertions, dispatching to different
    queues depending on the requested `ProvenanceStorageInterface` methods (16
    queues per methods). For instance, the `content` exchange handles all requests
    for `content_add` and `relation_add` for both relations `CNT_EARLY_IN_REV` and
    `CNT_IN_DIR` (ie. relations with content as source). In each case, requests
    are forwarded to 1 of 16 possible workers, depending on the sha1 id of the
    content.

commit 2a5fe87fbd76aa30e44ea7703a85d5a4b70e574c
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Fri Aug 20 12:21:27 2021 +0200

    Add new RabbitMQ-based client/server API
    
    Get methods in the `ProvenanceStorageInterface` are called through a server that
    guarantees conflict-free writings to the underlying database.
    
    Set methods are called directly from the client to avoid RCP overhead for reads.
    
    The server spawns multiple processes to handle independent requests concurrently.

commit 62884e23dd1164274fd89a09acedae8977a8e0f3
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date:   Thu Oct 14 12:03:47 2021 +0200

    Improve PostgreSQL storage scheme for the `with-path-denormalized` flavor
    
    Previous version was storing arrays of strings representing tuples for the
    denormalized relations (`dst` and `loc` of the relation resp.). While that
    simplified the check for duplicates, it turned out to be very inefficient
    in terms of disk usage. The new version has two distinct lists if `bigint`
    (ie. internal ids) for `dst` and `loc` resp. To check for duplicates the
    lists should be zipped, and repeated tuples filtered.

See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/464/ for more details.