This should be resolved now (actually, on the 31st at 16).

my understanding of the logs is that three OSDs (osd.3, osd.11, and osd.14 IIRC) were suddenly all stuck "waiting for subops". Unfortunately, while this is easy to debug while the issue is happening (dump_ops_in_flight), I didn't think of it at the time, given the hurry to fix the issue...

FTR I've tried to investigate a bit to find clues of what the origin of the outage was, but I did not find any obvious culprit.

(claiming i said ;)

(Claiming the task to find it back more easily through my activity view.)

Started in the same tmux session as the sourceforge ingestion.

It would probably make sense to set up a new worker instance for this to avoid interfering with the regular loading.

Smoke test it with local bitbucket repositories

that's next.

Smoke test it with remote repositories.

Thanks for the heads up @ both of you.

@ardumont noticed the replication was blocked, but our automated monitoring didn't alert us. He unblocked the replication, so your code should appear in the next hours.

This may simply be lag between the loader's and the frontend's databases.

Heads up, this is running slightly different now.

For history purpose readabillty, this must bev2.1 git-patched version (not a release per say).
A more recent version release which is a tag v2.1.0 [1] has been done built with the work
solving the work @olasd started with versioned extid.

Latest mercurial loader v2.1 deployed [1] [2]
We should be able to continue with this now.

It's working but the check does not pass green [1]. As far as i could tell, the unsuccessful
event [1] is seen as failure by the check.

Still ongoing.

17:31:19 softwareheritage-scheduler@belvedere:5432=> select now(), visit_type, count(*) from listed_origins lo inner join listers l on l.id=lo.lister_id where l.name='gitlab' and l.instance_name='gitlab' group by visit_type order by count(
*) desc;
+-------------------------------+------------+---------+
|              now              | visit_type |  count  |
+-------------------------------+------------+---------+
| 2021-07-29 15:46:55.979469+00 | git        | 1747880 |
+-------------------------------+------------+---------+
(1 row)

At the end of it all though, the final production check end-to-end for mercurial origin should go green.