Page MenuHomeSoftware Heritage

Update large workers to consume standard git queue
Closed, MigratedEdits Locked

Description

Those workers are currently used to dedicate some work load to decrease lag on
gitlab.com origins. The lag has subsided and those workers are currently less loaded.

So we can dedicate some work load to our standard git queue. Notably, the github origins
are lagging in term of consumption, making those workers consume from that queue should
help.

This is the task used to track the work to do so.

Plan:

  • D6873: make worker[17-18] swh-worker@loader_git consume from the standard git queue
  • D6873: (if load ok) increase concurrency for that service
  • D6873: decrease concurrency for swh-worker@loader_oneshot (which is less sensitive currently)
  • T3825#76379: potentially increase the max_queue_length, if the consumption becomes fast enough to drain the current 20k configured.

[1] Prior to this, the workers were used to ingest bitbucket mercurial and sourceforge
origins.

Event Timeline

ardumont triaged this task as Normal priority.Jan 3 2022, 12:36 PM
ardumont created this task.

potentially increase the max_queue_length (if the consumption becomes fast enough to
drain the current 20k configured)

The standard queue (filled with ~20k origins) is now completely consumed. It got bumped
to 40k [1].

[1]

12:28:41 softwareheritage-scheduler@belvedere:5432=> select * from task_type where type in ('load-git');
+----------+------------------------------+------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
|   type   |         description          |               backend_name               | default_interval | min_interval | max_interval | backoff_factor | max_queue_length | num_retries | retry_delay |
+----------+------------------------------+------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
| load-git | Update an origin of type git | swh.loader.git.tasks.UpdateGitRepository | 64 days          | 12:00:00     | 64 days      |              2 |            20000 |           3 | (null)      |
+----------+------------------------------+------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
(1 row)

12:29:06 softwareheritage-scheduler@belvedere:5432=> update task_type set max_queue_length=40000 where type in ('load-git');
UPDATE 1
Time: 11.346 ms

12:29:34 softwareheritage-scheduler@belvedere:5432=> select * from task_type where type in ('load-git');
+----------+------------------------------+------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
|   type   |         description          |               backend_name               | default_interval | min_interval | max_interval | backoff_factor | max_queue_length | num_retries | retry_delay |
+----------+------------------------------+------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
| load-git | Update an origin of type git | swh.loader.git.tasks.UpdateGitRepository | 64 days          | 12:00:00     | 64 days      |              2 |            40000 |           3 | (null)      |
+----------+------------------------------+------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
(1 row)
ardumont changed the task status from Open to Work in Progress.Jan 4 2022, 4:07 PM
ardumont moved this task from Backlog to in-progress on the System administration board.

It's back to 20k as it's enough now that we have a way forward in T3827.

The gist of this task is done btw so closing.

This graph [1] should make apparent (at some point) that the consumption is
now faster (for both github and gitlab.com origins).

[1] https://grafana.softwareheritage.org/goto/Ub_WEYAnk?orgId=1