Page MenuHomeSoftware Heritage
Create Task
Maniphest T3455

Make bitbucket origins ingestion concurrent
Closed, MigratedEdits Locked
Actions

Description

The current execution is running slowly for now. It's a simple loop over the mapping
file (one line is one origin) to trigger the ingestion per origin.

Make it go faster using the actual loader_oneshot.

For this, we need to iterate over the mapping file and send tasks
swh.loader.mercurial.tasks.LoadMercurial with the proper parameters.

That also have the following pros:

  • no need to deploy another special worker
  • have the systemctl logs pushed to elk, which make it browsable more easily (at least for staff) [1]

[1] http://kibana0.internal.softwareheritage.org:5601/goto/23e431d304850e45e602752e849a3f28

Related Objects
Search...

Event Timeline

ardumont renamed this task from Schedule properly the bitbucket origins to Make bitbucket origins ingestion go faster.Aug 3 2021, 10:05 AM
ardumont triaged this task as High priority.
ardumont created this task.
ardumont mentioned this in T3338: Load the archived bitbucket mercurial repositories.
ardumont renamed this task from Make bitbucket origins ingestion go faster to Make bitbucket origins ingestion concurrent.Aug 3 2021, 10:07 AM
ardumont updated the task description. (Show Details)Aug 3 2021, 10:20 AM
ardumont mentioned this in rDSNIP34ceae33c6d7: Adapt schedule script to allow bitbucket mercurial ingestion.Aug 5 2021, 12:27 PM
ardumont added a comment.EditedAug 5 2021, 12:32 PM

With the following change in the snippet code (check commit ^):

swhworker@worker17:~$ cat $base_dir/mapping-to-repos.txt |
while read dir url; do
  repo_dir="$base_dir/$dir";
  visit_date=`stat -c %z $repo_dir/.hg/blackbox.log | sed -E 's/ \+0000/+0000/'`;
  echo $url $repo_dir $visit_date;
done | SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_oneshot.yml \
  python3 schedule_with_queue_length_check.py --queue-name bitbucket-mercurial |
  tee -a scheduled-origins-bitbucket-20210805.txt

I've triggered back everything, including the 1k that were already run (mozilla-central
still ongoing so with the previous concurrency of 1, nothing bulged much since then).

That will also allow to have the systemd logs pushed to elasticsearch as well.

ardumont moved this task from Backlog to code-review/await-feedback/pause on the System administration board.Aug 5 2021, 12:33 PM
ardumont added a comment.EditedAug 5 2021, 12:44 PM

That will also allow to have the systemd logs pushed to elasticsearch as well.

This can be followed in the dedicated dashboard created for the occasion [1]

[1] http://kibana0.internal.softwareheritage.org:5601/goto/23e431d304850e45e602752e849a3f28

ardumont changed the task status from Open to Work in Progress.Aug 5 2021, 12:45 PM
ardumont updated the task description. (Show Details)Aug 5 2021, 12:47 PM
ardumont added a comment.Aug 6 2021, 9:17 AM

All messages are queued in the oneshot:swh.loader.mercurial.tasks.LoadArchive queue [1]
Those are concurrently ingested by the worker17.

[1] Detailed origins queued in swhworker@worker17:~/scheduled-origins-bitbucket-20210805.txt.gz

swhworker@worker17:~$ ls -lah scheduled-origins-bitbucket-20210805.txt.gz
-rw-r--r-- 1 swhworker swhworker 16M Aug  5 17:49 scheduled-origins-bitbucket-20210805.txt.gz
swhworker@worker17:~$ gzip -dc scheduled-origins-bitbucket-20210805.txt.gz - | head -1
{'args': [], 'kwargs': {'url': 'https://bitbucket.org/rarchk/jsmn', 'directory': '/srv/storage/space/mirrors/boatbucket/refugees/data/f1/f139ec4a-f546-470f-9c65-4bdc07d499dd/main-repo.mercurial', 'visit_date': '2020-04-30 19:32:05.02174075
ardumont closed this task as Resolved.Aug 6 2021, 9:22 AM
ardumont updated the task description. (Show Details)
ardumont moved this task from code-review/await-feedback/pause to done on the System administration board.Nov 18 2021, 3:17 PM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:03 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal