Details
Details
- Reviewers
jayeshv - Group Reviewers
Reviewers - Commits
- rDPROVa8009be206dc: Add StatsD support to `ArchiveInterface` implementations
Diff Detail
Diff Detail
- Repository
- rDPROV Provenance database
- Branch
- master
- Lint
Lint Skipped - Unit
Unit Tests Skipped - Build Status
Buildable 23614 Build 36850: Phabricator diff pipeline on jenkins Jenkins console · Jenkins Build 36849: arc lint + arc unit
Event Timeline
Comment Actions
Build is green
Patch application report for D6256 (id=22653)
Could not rebase; Attempt merge onto 2e997f3103...
Updating 2e997f3..6e5bd0c Fast-forward mypy.ini | 3 + swh/provenance/__init__.py | 14 +- swh/provenance/api/client.py | 528 ++++++++++++++++- swh/provenance/api/server.py | 784 ++++++++++++++++++++++--- swh/provenance/cli.py | 29 +- swh/provenance/interface.py | 4 +- swh/provenance/mongo/backend.py | 4 +- swh/provenance/postgresql/archive.py | 13 + swh/provenance/postgresql/provenance.py | 10 +- swh/provenance/storage/archive.py | 13 + swh/provenance/tests/conftest.py | 31 +- swh/provenance/tests/test_archive_interface.py | 16 +- 12 files changed, 1313 insertions(+), 136 deletions(-)
Changes applied before test
commit 6e5bd0c00d9778edc412779f75fa2f591b9dd838
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Sep 14 15:33:33 2021 +0200
Add StatsD support on `ArchiveInterface` implementations
commit 3bde39bfe1560cff91f4cd07a8cb37bf8d899091
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Sep 14 15:21:55 2021 +0200
Improve timeout while waiting for response handling on client side
commit 3aaa272b2e540de50d864979cade5538d0d30ca1
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Sep 14 14:50:42 2021 +0200
Switch to use a topic exchange instead of a direct one on remote backend
This allows to have less working processes (16 per entity and 16 extra for locations),
each one listening in several queues, depending on the requested insertion method.
All pending conflict resolutions is solved now.
commit 7f1b31b6975a350ad336f89564b32e4adaa79e09
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Sep 3 16:06:18 2021 +0200
Improve connection error handling on both client and server side
Change the RabbitMQ scheme to have 10 exchange (one per set method) and
16 routing keys per exchange, instead of the original 160 exchanges with
1 routing key each. Still 160 worker processes though.
commit f306903a932648f4837cd6365919fc22078c4dab
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Sep 1 11:27:02 2021 +0200
Remove old client/server storage based on `swh.core.api.RPCClient`
commit 1b856a23946860665341804a0eba97b461cbc97d
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Aug 31 13:36:34 2021 +0200
Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss
The workers now use `pika.SelectConnection` and make and explicit handle
of its life-cycle
commit a26fa8c35224f8157ad54e6ceabd5014086e60ce
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Sep 7 15:45:34 2021 +0200
Refactor server to use multiple sub-processes instead of threads
commit 17e67552ddf718e7c3652bce96fa76f17fbb5ce7
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Wed Aug 25 13:52:31 2021 +0200
Add support for `relation_add` to the RabbitMQ server
commit 5623696fd976cc34d5ae57da3da64f42e3ab3ce4
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Aug 24 15:09:02 2021 +0200
Remove get queues and have client read from `ProvenanceStorage` directly
Add a new queue to the server for clients to retrieve storage configuration
commit 95e7f73c9d52755cae98abbf07f93abbdf19909d
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Aug 24 11:45:47 2021 +0200
Split set method's requests into several queues on server side
commit 081303596f3e4daa5759f8dfafa28022b0d76a79
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Fri Aug 20 12:21:27 2021 +0200
Add new RabbitMQ-based client/server API
commit 3383cae57ef741859c33300045bc8a149dfc0d35
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Sep 14 09:58:20 2021 +0200
Generalize types for `content_add` and `directory_add` in the storage interfaceSee https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/340/ for more details.
Comment Actions
Build is green
Patch application report for D6256 (id=22681)
Rebasing onto 3383cae57e...
Current branch diff-target is up to date.
Changes applied before test
commit a8009be206dc1e414a69a708f826266b750b74cd
Author: Andres Ezequiel Viso <aeviso@softwareheritage.org>
Date: Tue Sep 14 15:33:33 2021 +0200
Add StatsD support to `ArchiveInterface` implementationsSee https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/343/ for more details.