make mypy happy

add missing parenthesis

update according the reviews

• simplify the cache management
• fix the doc strings

looks good (for what it's worth)

It's confirmed that the issue seems to be on the python part of the current implementation so I'm eager to see D7890 landed ;)

I reversed engineered the py4j communication protocol, so next time it will hang, we should be able to tell if the issue is on the gateway server side or on the python side:

• Create a name pipe

mkfifo /tmp/test
chmod a+w /tmp/test
tail -F /tmp/test

• query the graph

ss -ltp | grep java
<get the port number>
telnet localhost <port number>
c
o0
get_handler
s/tmp/test
e

Looks like something is wrong in the operator state management.
For what I found on internet, it could be related to the cert-manager version but it should be already fixed. For example: https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/issues/315
(The current cert-manager version in the cluster is 1.8.0)

A dozen of clients running in the provenance-client01 are using the multiplexer configuration.
It seems to work correctly

rebase

Update according the reviews

• Add and fix license headers
• Ensure the _revisions_count variable was computed before returning its value

\o/ well done

I've deliberately created the diff with the 3 commits inside, I just forgot to update the title ;)

fixed with the workaround in D7972

Because it looks simpler and faster to have this workaround here.

is it not possible to use a sql script directly, in the idea of [1] ?
I suppose it will make the script simpler as the postgresql image logic will take care of running the script only during the database initialization.

This issue was a false lead, different snapshots are declared for the same origin letting me believe there were some duplicates.
I completely missed it and fall head first into the trap

only use the postgresql paramater to configure the max connection

update the parameter count limit according the new supported ones

I've also added the permissions to the user mirror-test-rw to create and destroy topics. So you should be able to manage the swh.test.objects[_privileged] topics lifecycle

btw the credentials are pushed in the credential store

heh, it misses the last part of the task (the commands to manage the topics)

Changing the status to resolved.
@douardda don't hesitate to reopen if it's not working as expected

Permission of users should be ok:

• mirror-test-rw

root@getty:~# diff -U3 /usr/local/sbin/create_kafka_users_rocquencourt_staging.sh create_kafka_test_users_rw_rocquencourt_staging.sh 
--- /usr/local/sbin/create_kafka_users_rocquencourt_staging.sh	2022-01-21 16:57:22.076322616 +0000
+++ create_kafka_test_users_rw_rocquencourt_staging.sh	2022-06-03 13:02:03.497371791 +0000
@@ -56,15 +72,15 @@
 	--entity-type users \
 	--entity-name $username

Topics created:

storage1 /opt/kafka/bin% TOPICS="content directory extid metadata_authority metadata_fetcher origin origin_visit origin_visit_status raw_extrinsic_metadata release revision skipped_content snapshot"

after some ingestion time, it seems the first analysis is wrong:

(provenance) provenance-client01:~$ sort ~/origin-client.log |grep Processing | uniq -c  | sort -n | tail -n 10
      6 INFO:swh.provenance.origin:Processing origin https://bitbucket.org/360factors/workflow-conditions.git
      6 INFO:swh.provenance.origin:Processing origin https://bitbucket.org/4s/cda-document-viewer.git
      6 INFO:swh.provenance.origin:Processing origin https://bitbucket.org/4tic/goc.git
      6 INFO:swh.provenance.origin:Processing origin https://bitbucket.org/ALEXks/sapfor_2017.git
      6 INFO:swh.provenance.origin:Processing origin https://bitbucket.org/ASIWeb/testcomplete.git
      7 INFO:swh.provenance.origin:Processing origin https://anonhg.netbsd.org/xsrc-draft/
      7 INFO:swh.provenance.origin:Processing origin https://anonhg.netbsd.org/xsrc-public/
      8 INFO:swh.provenance.origin:Processing origin https://anonhg.netbsd.org/xsrc/
     13 INFO:swh.provenance.origin:Processing origin https://bitbucket.org/9front/plan9front
     20 INFO:swh.provenance.origin:Processing origin https://anongit.kde.org/kdenlive.git

It's not as bad as expected, it seems only 2 clients are proceeding the same origin at the same time:

I'm not sure if the service should be stopped before being removed but otherwise it looks ok

fix typos according to the feedback

after rereading this together, it seems the worker activation is missing

• Is the scheduler section in 'swh::deploy::indexer_journal_client::config' still needed ?

update storage default url to match a local service deployed on the cluster

The target of this repository is staging / production or even test environments.

Remove testing values file

An interesting lead that could possibly explain what happened on the cluster : https://etcd.io/docs/v3.4/faq/#should-i-add-a-member-before-removing-an-unhealthy-member

FI: an odd number of nodes is recommended for an etcd cluster

The cluster is up and running.