Page MenuHomeSoftware Heritage
Create Task
Maniphest T3825

Update large workers to consume standard git queue
Closed, MigratedEdits Locked
Actions

Description

Those workers are currently used to dedicate some work load to decrease lag on
gitlab.com origins. The lag has subsided and those workers are currently less loaded.

So we can dedicate some work load to our standard git queue. Notably, the github origins
are lagging in term of consumption, making those workers consume from that queue should
help.

This is the task used to track the work to do so.

Plan:

  • D6873: make worker[17-18] swh-worker@loader_git consume from the standard git queue
  • D6873: (if load ok) increase concurrency for that service
  • D6873: decrease concurrency for swh-worker@loader_oneshot (which is less sensitive currently)
  • T3825#76379: potentially increase the max_queue_length, if the consumption becomes fast enough to drain the current 20k configured.

[1] Prior to this, the workers were used to ingest bitbucket mercurial and sourceforge
origins.

Revisions and Commits

rSPSITE puppet-swh-site
D6873rSPSITEdd9193636941 worker-large: Make worker consume standard git queue

Related Objects

Event Timeline

ardumont triaged this task as Normal priority.Jan 3 2022, 12:36 PM
ardumont created this task.
ardumont added a revision: D6873: worker-large: Make worker consume standard git queue.Jan 3 2022, 12:41 PM
ardumont added a commit: rSPSITEdd9193636941: worker-large: Make worker consume standard git queue.Jan 3 2022, 2:21 PM
ardumont added a project: System administration.Jan 3 2022, 2:27 PM
ardumont added a comment.Jan 3 2022, 2:32 PM

potentially increase the max_queue_length (if the consumption becomes fast enough to
drain the current 20k configured)

The standard queue (filled with ~20k origins) is now completely consumed. It got bumped
to 40k [1].

[1]

12:28:41 softwareheritage-scheduler@belvedere:5432=> select * from task_type where type in ('load-git');
+----------+------------------------------+------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
|   type   |         description          |               backend_name               | default_interval | min_interval | max_interval | backoff_factor | max_queue_length | num_retries | retry_delay |
+----------+------------------------------+------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
| load-git | Update an origin of type git | swh.loader.git.tasks.UpdateGitRepository | 64 days          | 12:00:00     | 64 days      |              2 |            20000 |           3 | (null)      |
+----------+------------------------------+------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
(1 row)

12:29:06 softwareheritage-scheduler@belvedere:5432=> update task_type set max_queue_length=40000 where type in ('load-git');
UPDATE 1
Time: 11.346 ms

12:29:34 softwareheritage-scheduler@belvedere:5432=> select * from task_type where type in ('load-git');
+----------+------------------------------+------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
|   type   |         description          |               backend_name               | default_interval | min_interval | max_interval | backoff_factor | max_queue_length | num_retries | retry_delay |
+----------+------------------------------+------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
| load-git | Update an origin of type git | swh.loader.git.tasks.UpdateGitRepository | 64 days          | 12:00:00     | 64 days      |              2 |            40000 |           3 | (null)      |
+----------+------------------------------+------------------------------------------+------------------+--------------+--------------+----------------+------------------+-------------+-------------+
(1 row)
ardumont updated the task description. (Show Details)Jan 3 2022, 2:34 PM
ardumont mentioned this in T3827: Analyze low performance scheduling.Jan 3 2022, 5:39 PM
ardumont changed the task status from Open to Work in Progress.Jan 4 2022, 4:07 PM
ardumont moved this task from Backlog to in-progress on the System administration board.

It's back to 20k as it's enough now that we have a way forward in T3827.

ardumont added a comment.Jan 4 2022, 4:08 PM

The gist of this task is done btw so closing.

ardumont closed this task as Resolved.Jan 4 2022, 4:08 PM
ardumont claimed this task.
ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.
ardumont moved this task from deployed/landed/monitoring to Component upgrades on the System administration board.
ardumont edited projects, added System administration (Component upgrades); removed System administration.
ardumont moved this task from Component upgrades to done on the System administration board.
ardumont edited projects, added System administration; removed System administration (Component upgrades).
ardumont added a comment.Jan 4 2022, 5:15 PM

This graph [1] should make apparent (at some point) that the consumption is
now faster (for both github and gitlab.com origins).

[1] https://grafana.softwareheritage.org/goto/Ub_WEYAnk?orgId=1

ardumont mentioned this in rSPSITE89ed293e1dc0: worker-large: Make workers loader_oneshot consume standard git queue.Jan 6 2022, 10:14 AM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:05 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal