Currently running on swh's internal infrastructure workers.

Command used to trigger the production of tasks:

cat INDEX.shuffle.svndump | ./bin/list-svndump-urls | SWH_WORKER_INSTANCE=swh_loader_svn python3 -u -m swh.loader.svn.producer svn-archive --visit-date 'Tue, 3 May 2016 17:16:32 +0200'

where:

• INDEX.shuffle.svndump is the output sort -R /srv/storage/space/mirrors/code.google.com/sources/INDEX.svndump which lists the full path to svn dumps
• ./bin/list-svndump-urls is in charge of building the url to use and dumps both url and full path to svn dump (decorator as a pipe)
• swh.loader.svn.producer is part of our debian package swh-loader-svn in charge of producing the actual tasks to queue

An update on this, this is still work in progress.

status

~168.5k repositories to ingest out of 575k repositories.

This is already scheduled in the loader-svn queue.
This is in stand-by (cf. below).

issues

As we hit regularly the following issue:

1. out of ram -> 2. worker killed without possibilities to run the cleaning step -> 3. out of disk issues -> 4. at least one worker idle (the one without space disk) which consumes without doing anything useful the remaining queued jobs.

Note: Important implementation detail, loader-svn works on disk.

workaround in progress

For now, @olasd and I worked on provisioning vms on beaubourg (almost there) to make those disk workers (git-disk + svn) run there.

The hypervisor beaubourg being not used as much as it could and louvre quite the opposite.

This is a workaround for now as other tasks have higher priorities (i'll mention them back when they exist :)

It got restarted 2 weeks ago (Monday 18th September 2017).
It just finished (Monday 2nd October 2017).

T676 now kicks in.