Page MenuHomeSoftware Heritage

Mercurial loaderFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Recent Activity

Thu, Nov 18

ardumont moved T3455: Make bitbucket origins ingestion concurrent from code-review/monitoring to done on the System administration board.
Thu, Nov 18, 3:17 PM · System administration, Mercurial loader
ardumont moved T3338: Load the archived bitbucket mercurial repositories from deployed/landed to done on the System administration board.
Thu, Nov 18, 3:16 PM · System administration, Mercurial loader

Wed, Nov 10

ardumont closed T3338: Load the archived bitbucket mercurial repositories as Resolved.

Done now, closing.

Wed, Nov 10, 12:13 PM · System administration, Mercurial loader

Mon, Nov 8

ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.

Remains 1 origin still ongoing... almost there...

Mon, Nov 8, 4:44 PM · System administration, Mercurial loader

Oct 26 2021

ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.

Remains 3 origins still ongoing...

Oct 26 2021, 12:39 PM · System administration, Mercurial loader

Oct 20 2021

ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.

Remains 6 origins still ongoing...

Oct 20 2021, 12:22 PM · System administration, Mercurial loader
ardumont closed T3658: Reference bitbucket mercurial origins, a subtask of T3338: Load the archived bitbucket mercurial repositories, as Resolved.
Oct 20 2021, 12:16 PM · System administration, Mercurial loader
ardumont closed T3658: Reference bitbucket mercurial origins as Resolved.
Oct 20 2021, 12:16 PM · System administration, Mercurial loader
ardumont added a comment to T3658: Reference bitbucket mercurial origins.

I've opened T3674 to discuss how to properly reference origins that are not the output of listers.

Oct 20 2021, 12:16 PM · System administration, Mercurial loader
ardumont moved T3658: Reference bitbucket mercurial origins from in-progress to deployed/landed on the System administration board.
Oct 20 2021, 12:12 PM · System administration, Mercurial loader
ardumont added a comment to T3658: Reference bitbucket mercurial origins.

New webapp version deployed [1], we can see the mercurial origins referenced as a discontinued service there.

Oct 20 2021, 12:11 PM · System administration, Mercurial loader

Oct 18 2021

ardumont updated the task description for T3658: Reference bitbucket mercurial origins.
Oct 18 2021, 12:09 PM · System administration, Mercurial loader
ardumont updated the task description for T3658: Reference bitbucket mercurial origins.
Oct 18 2021, 12:09 PM · System administration, Mercurial loader

Oct 15 2021

ardumont added a comment to T3647: CloneFailure on non-existent hg repo hosted on phabricator.

Right, that is not cloneable [1].
So we need to handle this more properly and raise a NotFound.

Oct 15 2021, 3:54 PM · Mercurial loader
ardumont changed the status of T3658: Reference bitbucket mercurial origins, a subtask of T3338: Load the archived bitbucket mercurial repositories, from Open to Work in Progress.
Oct 15 2021, 9:49 AM · System administration, Mercurial loader
ardumont changed the status of T3658: Reference bitbucket mercurial origins from Open to Work in Progress.
Oct 15 2021, 9:49 AM · System administration, Mercurial loader
ardumont added a comment to T3658: Reference bitbucket mercurial origins.

A first simple solution has been implemented in the webapp for now [1].
It's not deployed yet.

Oct 15 2021, 9:48 AM · System administration, Mercurial loader
ardumont renamed T3658: Reference bitbucket mercurial origins from Reference bitbucket mercurial origins in scheduler metrics to Reference bitbucket mercurial origins.
Oct 15 2021, 9:45 AM · System administration, Mercurial loader

Oct 14 2021

ardumont added a revision to T3658: Reference bitbucket mercurial origins: D6475: Reference ingested but discontinued bitbucket mercurial origins.
Oct 14 2021, 3:42 PM · System administration, Mercurial loader
ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.

Actual count on bitbucket origins:

14:23:32 softwareheritage@belvedere:5432=> select now(), count(distinct url) from origin o inner join origin_visit ov on o.id=ov.origin where o.url like 'https://bitbucket.org/%' and ov.type='hg';
+-------------------------------+--------+
|              now              | count  |
+-------------------------------+--------+
| 2021-10-14 12:23:33.901117+00 | 336795 |
+-------------------------------+--------+
(1 row)
Oct 14 2021, 3:32 PM · System administration, Mercurial loader
olasd updated subscribers of T3658: Reference bitbucket mercurial origins.
In T3658#72284, @olasd wrote:

We could argue that adding a separate, "virtual" lister instance for these bulk archived origins would make sense, but I don't know if it's worth the bother.

Oct 14 2021, 3:20 PM · System administration, Mercurial loader
olasd added a comment to T3658: Reference bitbucket mercurial origins.

I was thinking of something ad-hoc such as:

Oct 14 2021, 3:17 PM · System administration, Mercurial loader
ardumont updated the task description for T3658: Reference bitbucket mercurial origins.
Oct 14 2021, 3:15 PM · System administration, Mercurial loader
ardumont updated the task description for T3658: Reference bitbucket mercurial origins.
Oct 14 2021, 3:13 PM · System administration, Mercurial loader
ardumont triaged T3658: Reference bitbucket mercurial origins as High priority.
Oct 14 2021, 3:07 PM · System administration, Mercurial loader
ardumont moved T3338: Load the archived bitbucket mercurial repositories from code-review/monitoring to deployed/landed on the System administration board.
Oct 14 2021, 3:00 PM · System administration, Mercurial loader
ardumont added a revision to T3338: Load the archived bitbucket mercurial repositories: D6474: changelog: Update bitbucket mercurial ingestion status.
Oct 14 2021, 2:30 PM · System administration, Mercurial loader

Oct 11 2021

vlorentz triaged T3647: CloneFailure on non-existent hg repo hosted on phabricator as Normal priority.
Oct 11 2021, 2:56 PM · Mercurial loader
ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.

The last origins are still ongoing. They are taking their time...

Oct 11 2021, 1:25 PM · System administration, Mercurial loader

Oct 7 2021

ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.

I'm gonna attend to this soon.

Oct 7 2021, 12:05 PM · System administration, Mercurial loader
ardumont added a comment to T3338: Load the archived bitbucket mercurial repositories.

A first run of bitbucket origins have been scheduled and mostly ingested now [1]
(remains only 13 large ones ongoing).

Oct 7 2021, 9:42 AM · System administration, Mercurial loader

Oct 4 2021

ardumont added a revision to T3612: Clean up mercurial loader code: D6404: Clean up missed unused module from early clean up.
Oct 4 2021, 5:46 PM · Mercurial loader

Sep 29 2021

ardumont closed T3612: Clean up mercurial loader code as Resolved.

Deployed.

Sep 29 2021, 9:00 AM · Mercurial loader

Sep 28 2021

ardumont added a revision to T3612: Clean up mercurial loader code: D6361: mercurial: Rename from_disk module to the main and official loader.
Sep 28 2021, 11:24 AM · Mercurial loader

Sep 27 2021

ardumont added a revision to T3612: Clean up mercurial loader code: D6360: mercurial: Drop legacy loader.
Sep 27 2021, 7:03 PM · Mercurial loader
ardumont triaged T3612: Clean up mercurial loader code as Normal priority.
Sep 27 2021, 7:02 PM · Mercurial loader

Sep 23 2021

ardumont closed T3584: loader mercurial edge case about missing mapping from revision to hgnode-id as Resolved.

Deployed v2.3.1 with that fix.

Sep 23 2021, 11:07 AM · Mercurial loader
ardumont added a revision to T3584: loader mercurial edge case about missing mapping from revision to hgnode-id: D6329: Fix branch bookmark id format so ingestion can finish.
Sep 23 2021, 10:14 AM · Mercurial loader
ardumont updated subscribers of T3584: loader mercurial edge case about missing mapping from revision to hgnode-id.

@Alphare any clues as to why the format here is not in sync? ^

Sep 23 2021, 10:13 AM · Mercurial loader
ardumont added a comment to T3584: loader mercurial edge case about missing mapping from revision to hgnode-id.

Ok, found where the wrong format is found somehow, the branching_info.bookmarks is not in the right format.

Sep 23 2021, 10:03 AM · Mercurial loader
ardumont added a comment to T3584: loader mercurial edge case about missing mapping from revision to hgnode-id.

The test helped, it's a mismatch format problem.
Uncomment the test, place the right pdb stanza in the code and behold:

Sep 23 2021, 9:50 AM · Mercurial loader
ardumont closed T3563: Analyze and make the bitbucket ingestion faster, a subtask of T3338: Load the archived bitbucket mercurial repositories, as Resolved.
Sep 23 2021, 9:13 AM · System administration, Mercurial loader
ardumont closed T3563: Analyze and make the bitbucket ingestion faster as Resolved.
Sep 23 2021, 9:13 AM · System administration, Mercurial loader
ardumont moved T3563: Analyze and make the bitbucket ingestion faster from code-review/monitoring to deployed/landed on the System administration board.
Sep 23 2021, 9:13 AM · System administration, Mercurial loader
ardumont added a comment to T3563: Analyze and make the bitbucket ingestion faster.

I've patched the systemd swh-worker@loader_oneshot to actually lift --autoscale 10,20

Sep 23 2021, 9:13 AM · System administration, Mercurial loader

Sep 22 2021

ardumont updated the task description for T3584: loader mercurial edge case about missing mapping from revision to hgnode-id.
Sep 22 2021, 10:39 AM · Mercurial loader

Sep 20 2021

ardumont added a comment to T3563: Analyze and make the bitbucket ingestion faster.

I've patched the systemd swh-worker@loader_oneshot to actually lift --autoscale 10,20
from celery cli. It's actually holding fine. And that coupled with the filtering server
side makes for a huge bump in speed. The archive db does not seem to mind at all.

Sep 20 2021, 6:13 PM · System administration, Mercurial loader
ardumont moved T3563: Analyze and make the bitbucket ingestion faster from in-progress to code-review/monitoring on the System administration board.
Sep 20 2021, 12:11 PM · System administration, Mercurial loader
ardumont added a comment to T3563: Analyze and make the bitbucket ingestion faster.
10:02:40 softwareheritage@belvedere:5432=> select now(), count(distinct url) from origin o inner join origin_visit ov on o.id=ov.origin where o.url like 'https://bitbucket.org/%' and ov.type='hg';
+------------------------------+--------+
|             now              | count  |
+------------------------------+--------+
| 2021-09-20 10:04:30.89072+00 | 280995 |
+------------------------------+--------+
(1 row)
Sep 20 2021, 12:11 PM · System administration, Mercurial loader
ardumont added a comment to T3563: Analyze and make the bitbucket ingestion faster.

Deployed the loader mercurial v2.3 (with filtering server side).
As expected, less time is spent in the method fetching the new changesets.

Sep 20 2021, 12:08 PM · System administration, Mercurial loader