This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

This task has been migrated to GitLab.

also, this should preferably be deployed after swh-indexer v2.7.0 so we don't need to reset indexer journal consumers to index metadata from Gitea, but not a requirement.

There's a few issues with the configuration of these indexer clients:
the traffic should not be going through the IPSec VPN. They need to use the public,
authenticated kafka endpoints. The IPSec load is making all azure communication
struggle.

ack, that should be "simple" enough to adapt [1]
[1] https://docs.softwareheritage.org/sysadm/mirror-operations/onboard.html?highlight=credential#how-to-create-the-journal-credentials

There's a few issues with the configuration of these indexer clients:

I'm guessing that's the extrinsic metadata indexer; others need to do plenty of random access to the storage, but that one consumes very quickly from Kafka. On the bright side, it consumes the entire topic within hours so parallelism could be reduced, as a quick fix

All the indexers were stopped at 20:00 FR because something was consummng all the bandwidth of the VPN between azure and the our infra.

root@pergamon:/etc/clustershell# clush -b -w @indexer-workers "puppet agent --disable 'stop indexer to avoid bandwith consumption'"
root@pergamon:/etc/clustershell# clush -b -w @indexer-workers "systemctl stop swh-indexer-journal-client@*"

There's a few issues with the configuration of these indexer clients:

The lag is subsiding now, slowly because only 1 journal client:

After further investigation w/ @vsellier, it's also related to our storage and indexer storage having too few gunicorn workers serving the journal clients (among other things).
So decreasing the batch size to something like 100 and fixing that should fairly help ^.

Activating debug log [1]

@vlorentz fixed another error directly within the model to deal with old versioned objects out of the model.
This meant a new release for swh.model and swh.indexer.

Unfortunately, now the indexer debian build is broken due to the objstorage debian build being broken...

objstorage build unstuck [1]
Triggered back the build for indexer.

[1] https://jenkins.softwareheritage.org/view/swh-debian%20(draft)/job/debian/job/packages/job/DOBJS/job/gbp-buildpackage/

@vlorentz fixed another error directly within the model to deal with old versioned objects out of the model.
This meant a new release for swh.model and swh.indexer.

Unfortunately, now the indexer debian build is broken due to the objstorage debian build being broken...

@vlorentz fixed another error directly within the model to deal with old versioned objects.
This meant a new release for swh.model and swh.indexer.

Drop the debian constraint on python3-rdflib. Trigger a rebuild and upgraded the package
again. Added the unconditional dependency on python3-rdflib-jsonld dependency (which on
latest debian release is not useful but without being a blocker).

Workers refuse to upgrade to the actual 2.4.3 version [1]. I did not realize my previous
upgrade from yesterday stopped at the v2.3.0.

Consumer lag is steadily increasing since yesterday [1]. I believe workers are hit by [2] issue.
I've opened [3] to try and unstuck it.

closing as T4459 will take care of this