The last origins are still ongoing. They are taking their time...

I'm gonna attend to this soon.

A first run of bitbucket origins have been scheduled and mostly ingested now [1]
(remains only 13 large ones ongoing).

Deployed.

Deployed v2.3.1 with that fix.

@Alphare any clues as to why the format here is not in sync? ^

Ok, found where the wrong format is found somehow, the branching_info.bookmarks is not in the right format.

The test helped, it's a mismatch format problem.
Uncomment the test, place the right pdb stanza in the code and behold:

I've patched the systemd swh-worker@loader_oneshot to actually lift --autoscale 10,20

I've patched the systemd swh-worker@loader_oneshot to actually lift --autoscale 10,20
from celery cli. It's actually holding fine. And that coupled with the filtering server
side makes for a huge bump in speed. The archive db does not seem to mind at all.

Deployed the loader mercurial v2.3 (with filtering server side).
As expected, less time is spent in the method fetching the new changesets.

Now most of the time can be spent in reading the actual mapping extids -> hgnode-id [1] to filter on something we already see.
Which does not change much from actual visits which already ended up in snapshot.
However that changes a lot for visits on forks where we can bypass already done work on those forks.

Actually restarted the loader_oneshot which now makes usage of the latest v2.2.0 loader mercurial.

Another run in on a large repository (which cannot finish, the error is independent
though) [1]

Packaged within the v0.37 version.
Deployed both in staging and production.

Deployed on staging and everything looked good.
So deployed on production as well.

First round of checks validates the new diff to make the current loader mercurial faster.

By the way, i forgot:

New day, new datapoint [1]

^ Temporarily disabled puppet agent and bumped the concurrency for that worker to 10 (around the time of the previous comment).

Ongoing ingestion is rather slow [1]
Because like for git origins, we can't know in advance rather large repositories.
So sometimes, ingestion appears stuck because we are dealing with large repositories (more than 2 hours of loading [2]).

It's currently ongoing on some large repositories (and some other large sourceforge svn repository).

It's currently running (data point ongoing...).

So this ingestion got stopped or crashed, at some point.
Probably around the db outage from last week (which emptied the rabbitmq queue).

All messages are queued in the oneshot:swh.loader.mercurial.tasks.LoadArchive queue [1]
Those are concurrently ingested by the worker17.

Concurrent deployment is ongoing ^ so this should go faster now, datapoint for later.
(mozilla-central fork from previous comment still ongoing...)

That will also allow to have the systemd logs pushed to elasticsearch as well.

With the following change in the snippet code (check commit ^):

Still progressing. It's not fast though since it's deployed rather simply. It's doing
one origin at a time which can end up seemingly stuck behind a big repository (currently
[1]):

(claiming i said ;)

(Claiming the task to find it back more easily through my activity view.)

Started in the same tmux session [2] as the sourceforge ingestion [1]

It would probably make sense to set up a new worker instance for this to avoid interfering with the regular loading.

Smoke test it with local bitbucket repositories

that's next.

Smoke test it with remote repositories.

For history purpose readabillty, this must bev2.1 git-patched version (not a release per say).
A more recent version release which is a tag v2.1.0 [1] has been done built with the work
solving the extid version inconsistency issue @olasd started.

Latest mercurial loader v2.1 deployed [1] [2]
We should be able to continue with this now.

Deployed \o/
Closing.

It's working but the check does not pass green [1]. As far as i could tell, the unsuccessful
event [1] is seen as failure by the check.

At the end of it all though, the final production check end-to-end for mercurial origin should go green.