Details
Details
- Reviewers
jayeshv - Group Reviewers
Reviewers - Commits
- rDPROVa8009be206dc: Add StatsD support to `ArchiveInterface` implementations
Diff Detail
Diff Detail
- Repository
- rDPROV Provenance database
- Branch
- master
- Lint
Lint Skipped - Unit
Unit Tests Skipped - Build Status
Buildable 23589 Build 36804: Phabricator diff pipeline on jenkins Jenkins console · Jenkins Build 36803: arc lint + arc unit
Event Timeline
Comment Actions
Build is green
Patch application report for D6256 (id=22653)
Could not rebase; Attempt merge onto 2e997f3103...
Updating 2e997f3..6e5bd0c Fast-forward mypy.ini | 3 + swh/provenance/__init__.py | 14 +- swh/provenance/api/client.py | 528 ++++++++++++++++- swh/provenance/api/server.py | 784 ++++++++++++++++++++++--- swh/provenance/cli.py | 29 +- swh/provenance/interface.py | 4 +- swh/provenance/mongo/backend.py | 4 +- swh/provenance/postgresql/archive.py | 13 + swh/provenance/postgresql/provenance.py | 10 +- swh/provenance/storage/archive.py | 13 + swh/provenance/tests/conftest.py | 31 +- swh/provenance/tests/test_archive_interface.py | 16 +- 12 files changed, 1313 insertions(+), 136 deletions(-)
Changes applied before test
commit 6e5bd0c00d9778edc412779f75fa2f591b9dd838 Author: Andres Ezequiel Viso <aeviso@softwareheritage.org> Date: Tue Sep 14 15:33:33 2021 +0200 Add StatsD support on `ArchiveInterface` implementations commit 3bde39bfe1560cff91f4cd07a8cb37bf8d899091 Author: Andres Ezequiel Viso <aeviso@softwareheritage.org> Date: Tue Sep 14 15:21:55 2021 +0200 Improve timeout while waiting for response handling on client side commit 3aaa272b2e540de50d864979cade5538d0d30ca1 Author: Andres Ezequiel Viso <aeviso@softwareheritage.org> Date: Tue Sep 14 14:50:42 2021 +0200 Switch to use a topic exchange instead of a direct one on remote backend This allows to have less working processes (16 per entity and 16 extra for locations), each one listening in several queues, depending on the requested insertion method. All pending conflict resolutions is solved now. commit 7f1b31b6975a350ad336f89564b32e4adaa79e09 Author: Andres Ezequiel Viso <aeviso@softwareheritage.org> Date: Fri Sep 3 16:06:18 2021 +0200 Improve connection error handling on both client and server side Change the RabbitMQ scheme to have 10 exchange (one per set method) and 16 routing keys per exchange, instead of the original 160 exchanges with 1 routing key each. Still 160 worker processes though. commit f306903a932648f4837cd6365919fc22078c4dab Author: Andres Ezequiel Viso <aeviso@softwareheritage.org> Date: Wed Sep 1 11:27:02 2021 +0200 Remove old client/server storage based on `swh.core.api.RPCClient` commit 1b856a23946860665341804a0eba97b461cbc97d Author: Andres Ezequiel Viso <aeviso@softwareheritage.org> Date: Tue Aug 31 13:36:34 2021 +0200 Rework `ProvenanceStorageRabbitMQWorker` to handle connection loss The workers now use `pika.SelectConnection` and make and explicit handle of its life-cycle commit a26fa8c35224f8157ad54e6ceabd5014086e60ce Author: Andres Ezequiel Viso <aeviso@softwareheritage.org> Date: Tue Sep 7 15:45:34 2021 +0200 Refactor server to use multiple sub-processes instead of threads commit 17e67552ddf718e7c3652bce96fa76f17fbb5ce7 Author: Andres Ezequiel Viso <aeviso@softwareheritage.org> Date: Wed Aug 25 13:52:31 2021 +0200 Add support for `relation_add` to the RabbitMQ server commit 5623696fd976cc34d5ae57da3da64f42e3ab3ce4 Author: Andres Ezequiel Viso <aeviso@softwareheritage.org> Date: Tue Aug 24 15:09:02 2021 +0200 Remove get queues and have client read from `ProvenanceStorage` directly Add a new queue to the server for clients to retrieve storage configuration commit 95e7f73c9d52755cae98abbf07f93abbdf19909d Author: Andres Ezequiel Viso <aeviso@softwareheritage.org> Date: Tue Aug 24 11:45:47 2021 +0200 Split set method's requests into several queues on server side commit 081303596f3e4daa5759f8dfafa28022b0d76a79 Author: Andres Ezequiel Viso <aeviso@softwareheritage.org> Date: Fri Aug 20 12:21:27 2021 +0200 Add new RabbitMQ-based client/server API commit 3383cae57ef741859c33300045bc8a149dfc0d35 Author: Andres Ezequiel Viso <aeviso@softwareheritage.org> Date: Tue Sep 14 09:58:20 2021 +0200 Generalize types for `content_add` and `directory_add` in the storage interface
See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/340/ for more details.
Comment Actions
Build is green
Patch application report for D6256 (id=22681)
Rebasing onto 3383cae57e...
Current branch diff-target is up to date.
Changes applied before test
commit a8009be206dc1e414a69a708f826266b750b74cd Author: Andres Ezequiel Viso <aeviso@softwareheritage.org> Date: Tue Sep 14 15:33:33 2021 +0200 Add StatsD support to `ArchiveInterface` implementations
See https://jenkins.softwareheritage.org/job/DPROV/job/tests-on-diff/343/ for more details.