Change Details

Those workers are currently used to dedicate some work load to decrease lag on gitlab.com origins. The lag has subsided and those workers are currently less loaded. So we can dedicate some work load to our standard git queue. Notably, the github origins are lagging in term of consumption, making those workers consume from that queue should help. This is the task used to track the work to do so. Plan: - make worker[17-18] swh-worker@loader_git consume from the standard git queue - (if load ok) increase concurrency for that service - decrease concurrency for swh-worker@loader_oneshot (which is less sensitive currently) - potentially increase the max_queue_length (if the consumption becomes fast enough to drain the current 20k configured) [1] prior to this, they were used to ingest sourceforge and bitbucket hg origins.

Those workers are currently used to dedicate some work load to decrease lag on gitlab.com origins. The lag has subsided and those workers are currently less loaded. So we can dedicate some work load to our standard git queue. Notably, the github origins are lagging in term of consumption, making those workers consume from that queue should help. This is the task used to track the work to do so. Plan: - [x] D6873: make worker[17-18] swh-worker@loader_git consume from the standard git queue - [x] D6873: (if load ok) increase concurrency for that service - [x] D6873: decrease concurrency for swh-worker@loader_oneshot (which is less sensitive currently) - [x] T3825#76379: potentially increase the max_queue_length, if the consumption becomes fast enough to drain the current 20k configured. [1] Prior to this, the workers were used to ingest bitbucket mercurial and sourceforge origins.

Those workers are currently used to dedicate some work load to decrease lag on gitlab.com origins. The lag has subsided and those workers are currently less loaded. So we can dedicate some work load to our standard git queue. Notably, the github origins are lagging in term of consumption, making those workers consume from that queue should help. This is the task used to track the work to do so. Plan: - [x] D6873: make worker[17-18] swh-worker@loader_git consume from the standard git - make worker[17-18] swh-worker@loader_git consume from the standard git queue - [x] D6873: (if load ok) increase concurrency for that service - [x] D6873: decrease concurrency for swh-worker@loader_oneshot (which is less sensitive currently) - currently) - [x] T3825#76379: potentially increase the max_queue_length (, if the consumption becomes fast enough tos fast enough to drain the current 20k configured). [1] p1] Prior to this, theythe workers were used to ingest sourceforgebitbucket mercurial and bitbucket hg sourceforge origins.