Page MenuHomeSoftware Heritage
Create Task
Maniphest T3581

List heptapod instance foss.heptapod.net
Closed, MigratedEdits Locked
Actions

Description

I've asked @marmoute and @Alphare on irc.

It's a somewhat ehhanced gitlab instance which lists 3 types of vcs repository types:

  • git
  • hg: native mercurial
  • hg-git: those should be converted in a few months by upstream, they can be ingested as mercurial repository safely

In practice, that means:

  • D6285: Adapt lister's current implementation to deal with other types [1]
  • D6286: Allow hg_git origns to be ingested as hg origins
  • D6287: Allow lister name to be something else than "gitlab"
  • Schedule a new gitlab instance named "heptapod" [3], instance="foss.heptapod.net" once ^ is packaged and deployed.
  • D6320: Update the archive logs about the finalized ingestion

[1] @anlambert did it in P1168, i iterated over it and packaged

[2] "swh-scheduler" db, "listers" table, "name" column, other gitlab instances have their name "gitlab"

[3] name must be different than "gitlab" to dedicate their coverage stats

Revisions and Commits

rDLS Listers
AbandonedD6394 gitlab: Allow listing of instances providing multiple vcs_type [PLEASE DELETE ME]
D6287rDLSc803fc2b5908 Allow gitlab lister's name to be overriden by task arguments
D6286rDLSfdb420238cc8 gitlab: Allow ingestion of hg_git origins as hg ones
D6285rDLS4e4edee478ce gitlab: Allow listing of instances providing multiple vcs_type
rDDOC Development documentation
D6320rDDOC4610129e4ed7 docs: Update archive changelog about the foss.heptpod.net instance

Related Objects
Search...

Event Timeline

ardumont triaged this task as Normal priority.Sep 16 2021, 5:38 PM
ardumont created this task.
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)Sep 16 2021, 5:44 PM
ardumont renamed this task from List "gitlab" instance foss.heptapod.net to List "gitlab-like" instance foss.heptapod.net.Sep 16 2021, 5:52 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont added a subscriber: anlambert.
ardumont added a revision: D6285: gitlab: Allow listing of instances providing multiple vcs_type.Sep 16 2021, 6:38 PM
ardumont added a commit: rDLS4e4edee478ce: gitlab: Allow listing of instances providing multiple vcs_type.Sep 17 2021, 10:50 AM
ardumont updated the task description. (Show Details)Sep 17 2021, 10:56 AM
marmoute added a comment.Sep 17 2021, 11:31 AM

The hg-git type are served as regular Mercurial repository. So they can be listed as Mercurial repository safely

ardumont added a comment.Sep 17 2021, 11:34 AM

Thanks for the heads up, gonna simplify stuff then.

ardumont updated the task description. (Show Details)Sep 17 2021, 11:34 AM
ardumont added a revision: D6286: gitlab: Allow ingestion of hg_git origins as hg ones.Sep 17 2021, 11:51 AM
ardumont added a commit: rDLSfdb420238cc8: gitlab: Allow ingestion of hg_git origins as hg ones.Sep 17 2021, 1:37 PM
ardumont added a revision: D6287: Allow gitlab lister's name to be overriden by task arguments.Sep 17 2021, 2:29 PM
ardumont updated the task description. (Show Details)Sep 17 2021, 2:36 PM
ardumont updated the task description. (Show Details)
ardumont added a commit: rDLSc803fc2b5908: Allow gitlab lister's name to be overriden by task arguments.Sep 17 2021, 3:43 PM
ardumont updated the task description. (Show Details)Sep 17 2021, 5:29 PM
ardumont changed the task status from Open to Work in Progress.Sep 22 2021, 9:50 AM
ardumont added a project: System administration.
ardumont moved this task from Backlog to Weekly backlog on the System administration board.
ardumont moved this task from Weekly backlog to in-progress on the System administration board.
ardumont updated the task description. (Show Details)Sep 22 2021, 10:14 AM
ardumont mentioned this in rSPSITEcf6550783c19: Drop no longer needed configuration option..Sep 22 2021, 10:29 AM
ardumont added a comment.EditedSep 22 2021, 2:06 PM

Status on this:

  • Added the foss heptapod instance for listing [1]
  • Ensure the run went smoothly [2]
  • Stop 12 some workers, keep only 4 for the instance is not harassed [3]
  • Trigger those new origins for ingestion
  • when done (or almost), starts back the other workers [4]
  • Update the archive logs when ingestion is done

[1]

swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ task add list-gitlab-incremental url="https://foss.heptapod.net/api/v4/" name=heptapod

[2]

10:02:35 softwareheritage-scheduler@belvedere:5432=> select now(), count(*) from listed_origins lo inner join listers l on lo.lister_id=l.id where l.name='heptapod';
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2021-09-22 08:05:33.014262+00 |   346 |
+-------------------------------+-------+
(1 row)

Time: 76.625 ms

[3] We do not have recurring mercurial forges yet so it's not a big deal. And save code
now have their own dedicated workers.

[4]

14:00:44 softwareheritage@belvedere:5432=> select now(), ov.type, count(distinct url) from origin o inner join origin_visit ov on o.id=ov.origin where o.url like 'https://foss.heptapod.net/%' and ov.type in ('hg', 'git') group by ov.type;
+-------------------------------+------+-------+
|              now              | type | count |
+-------------------------------+------+-------+
| 2021-09-22 12:04:29.419932+00 | git  |     8 |
| 2021-09-22 12:04:29.419932+00 | hg   |   343 |
+-------------------------------+------+-------+
(2 rows)

Time: 50896.112 ms (00:50.896)
ardumont updated the task description. (Show Details)Sep 22 2021, 2:06 PM
ardumont added a revision: D6320: docs: Update archive changelog about the foss.heptpod.net instance.Sep 22 2021, 2:09 PM
ardumont updated the task description. (Show Details)Sep 22 2021, 2:09 PM
ardumont claimed this task.Sep 22 2021, 2:14 PM
ardumont added a commit: rDDOC4610129e4ed7: docs: Update archive changelog about the foss.heptpod.net instance.Sep 22 2021, 2:32 PM
ardumont closed this task as Resolved.Sep 22 2021, 2:32 PM
ardumont updated the task description. (Show Details)
ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.
ardumont mentioned this in T3597: List heptapod instance https://forge.extranet.logilab.fr/.Sep 22 2021, 2:36 PM
ardumont renamed this task from List "gitlab-like" instance foss.heptapod.net to List heptapod instance foss.heptapod.net.Sep 23 2021, 8:53 AM
ardumont added a parent task: T3599: List and ingest heptapod instances.
zack added a project: Archive coverage.Sep 23 2021, 11:32 AM
borisbaldassari added a revision: D6394: gitlab: Allow listing of instances providing multiple vcs_type [PLEASE DELETE ME].Oct 2 2021, 9:06 PM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:04 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal