Page MenuHomeSoftware Heritage

List heptapod instance foss.heptapod.net
Closed, ResolvedPublic

Description

I've asked @marmoute and @Alphare on irc.

It's a somewhat ehhanced gitlab instance which lists 3 types of vcs repository types:

  • git
  • hg: native mercurial
  • hg-git: those should be converted in a few months by upstream, they can be ingested as mercurial repository safely

In practice, that means:

  • D6285: Adapt lister's current implementation to deal with other types [1]
  • D6286: Allow hg_git origns to be ingested as hg origins
  • D6287: Allow lister name to be something else than "gitlab"
  • Schedule a new gitlab instance named "heptapod" [3], instance="foss.heptapod.net" once ^ is packaged and deployed.
  • D6320: Update the archive logs about the finalized ingestion

[1] @anlambert did it in P1168, i iterated over it and packaged

[2] "swh-scheduler" db, "listers" table, "name" column, other gitlab instances have their name "gitlab"

[3] name must be different than "gitlab" to dedicate their coverage stats

Event Timeline

ardumont triaged this task as Normal priority.Sep 16 2021, 5:38 PM
ardumont created this task.
ardumont updated the task description. (Show Details)
ardumont renamed this task from List "gitlab" instance foss.heptapod.net to List "gitlab-like" instance foss.heptapod.net.Sep 16 2021, 5:52 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont added a subscriber: anlambert.

The hg-git type are served as regular Mercurial repository. So they can be listed as Mercurial repository safely

Thanks for the heads up, gonna simplify stuff then.

ardumont updated the task description. (Show Details)
ardumont changed the task status from Open to Work in Progress.Sep 22 2021, 9:50 AM
ardumont moved this task from Backlog to Weekly backlog on the System administration board.
ardumont moved this task from Weekly backlog to in-progress on the System administration board.

Status on this:

  • Added the foss heptapod instance for listing [1]
  • Ensure the run went smoothly [2]
  • Stop 12 some workers, keep only 4 for the instance is not harassed [3]
  • Trigger those new origins for ingestion
  • when done (or almost), starts back the other workers [4]
  • Update the archive logs when ingestion is done

[1]

swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ task add list-gitlab-incremental url="https://foss.heptapod.net/api/v4/" name=heptapod

[2]

10:02:35 softwareheritage-scheduler@belvedere:5432=> select now(), count(*) from listed_origins lo inner join listers l on lo.lister_id=l.id where l.name='heptapod';
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2021-09-22 08:05:33.014262+00 |   346 |
+-------------------------------+-------+
(1 row)

Time: 76.625 ms

[3] We do not have recurring mercurial forges yet so it's not a big deal. And save code
now have their own dedicated workers.

[4]

14:00:44 softwareheritage@belvedere:5432=> select now(), ov.type, count(distinct url) from origin o inner join origin_visit ov on o.id=ov.origin where o.url like 'https://foss.heptapod.net/%' and ov.type in ('hg', 'git') group by ov.type;
+-------------------------------+------+-------+
|              now              | type | count |
+-------------------------------+------+-------+
| 2021-09-22 12:04:29.419932+00 | git  |     8 |
| 2021-09-22 12:04:29.419932+00 | hg   |   343 |
+-------------------------------+------+-------+
(2 rows)

Time: 50896.112 ms (00:50.896)
ardumont updated the task description. (Show Details)
ardumont moved this task from in-progress to deployed/landed on the System administration board.
ardumont moved this task from deployed/landed to done on the System administration board.
ardumont renamed this task from List "gitlab-like" instance foss.heptapod.net to List heptapod instance foss.heptapod.net.Sep 23 2021, 8:53 AM