Done now, closing.

Remains 1 origin still ongoing... almost there...

Remains 3 origins still ongoing...

Remains 6 origins still ongoing...

I've opened T3674 to discuss how to properly reference origins that are not the output of listers.

New webapp version deployed [1], we can see the mercurial origins referenced as a discontinued service there.

Right, that is not cloneable [1].
So we need to handle this more properly and raise a NotFound.

A first simple solution has been implemented in the webapp for now [1].
It's not deployed yet.

Actual count on bitbucket origins:

14:23:32 softwareheritage@belvedere:5432=> select now(), count(distinct url) from origin o inner join origin_visit ov on o.id=ov.origin where o.url like 'https://bitbucket.org/%' and ov.type='hg';
+-------------------------------+--------+
|              now              | count  |
+-------------------------------+--------+
| 2021-10-14 12:23:33.901117+00 | 336795 |
+-------------------------------+--------+
(1 row)

In T3658#72284, @olasd wrote:

We could argue that adding a separate, "virtual" lister instance for these bulk archived origins would make sense, but I don't know if it's worth the bother.

I was thinking of something ad-hoc such as:

The last origins are still ongoing. They are taking their time...

I'm gonna attend to this soon.

A first run of bitbucket origins have been scheduled and mostly ingested now [1]
(remains only 13 large ones ongoing).

Deployed.

Deployed v2.3.1 with that fix.

@Alphare any clues as to why the format here is not in sync? ^

Ok, found where the wrong format is found somehow, the branching_info.bookmarks is not in the right format.

The test helped, it's a mismatch format problem.
Uncomment the test, place the right pdb stanza in the code and behold:

I've patched the systemd swh-worker@loader_oneshot to actually lift --autoscale 10,20

I've patched the systemd swh-worker@loader_oneshot to actually lift --autoscale 10,20
from celery cli. It's actually holding fine. And that coupled with the filtering server
side makes for a huge bump in speed. The archive db does not seem to mind at all.

Deployed the loader mercurial v2.3 (with filtering server side).
As expected, less time is spent in the method fetching the new changesets.

Now most of the time can be spent in reading the actual mapping extids -> hgnode-id [1] to filter on something we already see.
Which does not change much from actual visits which already ended up in snapshot.
However that changes a lot for visits on forks where we can bypass already done work on those forks.

Actually restarted the loader_oneshot which now makes usage of the latest v2.2.0 loader mercurial.

Another run in on a large repository (which cannot finish, the error is independent
though) [1]

Packaged within the v0.37 version.
Deployed both in staging and production.

Deployed on staging and everything looked good.
So deployed on production as well.

First round of checks validates the new diff to make the current loader mercurial faster.

By the way, i forgot:

New day, new datapoint [1]

^ Temporarily disabled puppet agent and bumped the concurrency for that worker to 10 (around the time of the previous comment).

Ongoing ingestion is rather slow [1]
Because like for git origins, we can't know in advance rather large repositories.
So sometimes, ingestion appears stuck because we are dealing with large repositories (more than 2 hours of loading [2]).

It's currently ongoing on some large repositories (and some other large sourceforge svn repository).

It's currently running (data point ongoing...).

So this ingestion got stopped or crashed, at some point.
Probably around the db outage from last week (which emptied the rabbitmq queue).

All messages are queued in the oneshot:swh.loader.mercurial.tasks.LoadArchive queue [1]
Those are concurrently ingested by the worker17.

Concurrent deployment is ongoing ^ so this should go faster now, datapoint for later.
(mozilla-central fork from previous comment still ongoing...)

That will also allow to have the systemd logs pushed to elasticsearch as well.

With the following change in the snippet code (check commit ^):

Still progressing. It's not fast though since it's deployed rather simply. It's doing
one origin at a time which can end up seemingly stuck behind a big repository (currently
[1]):

(claiming i said ;)

(Claiming the task to find it back more easily through my activity view.)

Started in the same tmux session [2] as the sourceforge ingestion [1]