Page MenuHomeSoftware Heritage
Create Task
Maniphest T1538

Add "forge" now
Closed, MigratedEdits Locked
Actions

Description

We want to streamline the process to add a new code hosting platform to our regular
crawling queue, similarly to the currently available save code now feature. In a
nutshell:

  • a user-facing form that allow any user to submit the URL of a new code hosting platform, specifying the technology used on that platform (this is similar to the interface used for the "Save code now" feature)
  • the request goes into a moderation queue for human review through an admin interface similar to the one for Save Code Now
  • if accepted, the process continues as described in T1537.

We need to think about the name we decide to use for this feature: "Save Forge Now" is
appealing but misleading, as it will take time to actually save the full state of the
forge (it will not be "now"), and we actually do more, as this is not a one shot
operation, but an addition to the regular crawling. "Add Forge Now" would correspond
more to the reality.

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT3972 bootstrap a d.s.o/user end user documentation page
 Migratedgitlab-migrationT4245 user docs:: write content for "Getting to know the archive" section
 Migratedgitlab-migrationT2197 Ingestion / coverage
 Migratedgitlab-migrationT2199 "Save Forge Now"
 Migratedgitlab-migrationT1538 Add "forge" now
 Migratedgitlab-migrationT4047 User interface to submit and follow Add forge now requests
 Migratedgitlab-migrationT4053 [Add forge now] Fix support of browsers with Javascript disabled
 Migratedgitlab-migrationT4051 Add forge now / Save code now: tabs are not preserved with the browser history
 Migratedgitlab-migrationT3998 [Add forge now] Change request status
 Migratedgitlab-migrationT3991 [Add forge now] Create moderation dashboard
 Migratedgitlab-migrationT4071 add forge now - add columns to list of requests
 Migratedgitlab-migrationT4072 add forge now - whole request list row clickable
 Migratedgitlab-migrationT3999 [Add forge now] Design a feedback tracking solution
 Migratedgitlab-migrationT3992 [Add forge now] Create request dashboard w/ basic info
 Migratedgitlab-migrationT4068 [add forge now] display request comment in history
Migratedgitlab-migrationT4001 [Add forge now] Display detailled log/history
 Migratedgitlab-migrationT3993 [Add forge now] Send submission mail to forge
 Migratedgitlab-migrationT4005 [Add forge now] Retrieve first repo archived info
 Migratedgitlab-migrationT4004 [Add forge now] Retrieve first listing info
 Migratedgitlab-migrationT4002 [Add forge now] Display messages
 Migratedgitlab-migrationT3733 Develop a "Add forge now" mock-up
 Migratedgitlab-migrationT4040 [Add forge now] Create a feature flag
 Migratedgitlab-migrationT4070 add forge now submission form
 Migratedgitlab-migrationT4069 [add forge now] add editable name and email fields if consent
 Migratedgitlab-migrationT3987 [Add forge now] Create Submission form w/basic info
 Migratedgitlab-migrationT4012 [Add forge now] Create a message panel for non logged user
 Migratedgitlab-migrationT4067 [add forge now] - change login link on submit form
Migratedgitlab-migrationT4003 [Add forge now] Add request comment
 Migratedgitlab-migrationT3989 [Add forge now] Create "browse add requests" tab
 Migratedgitlab-migrationT4006 [Add forge now] Display link to search results for the scheduled request
 Migratedgitlab-migrationT4011 [Add forge now] Filter user requests
 Migratedgitlab-migrationT4091 [Add forge now] Display extra information when users list their requests
 Migratedgitlab-migrationT4073 add tooltip on add forge now request status
Migratedgitlab-migrationT4078 add forge now - add a link on forge url
 Migratedgitlab-migrationT3988 [Add forge now] Create Help text tab
 Migratedgitlab-migrationT3990 [Add forge now] Implement "Submit request" API feature
 Migratedgitlab-migrationT4050 [Add forge now] Refreshing the page loses the current tab
 Migratedgitlab-migrationT4015 [Add forge now] Send the submitter a receipt email
 Migratedgitlab-migrationT4075 add forge now - refactor help content
 Migratedgitlab-migrationT4077 add forge now - client-side forge url check
 Migratedgitlab-migrationT4074 add forge now - simplify date format display
 Migratedgitlab-migrationT4076 add forge now - use human readable status names
 Migratedgitlab-migrationT4058 Add forge now request processing admin tools
 Migratedgitlab-migrationT1537 semi-automated addition of new "forges"
 Migratedgitlab-migrationT4017 [Add forge now] Add an accepted request to the scheduling process
 Migratedgitlab-migrationT4016 [Add forge now] Notify first listing done to submitter
Migratedgitlab-migrationT4059 Add forge now documentation
 Migratedgitlab-migrationT4038 [Add forge now] Document new feature
 Migratedgitlab-migrationT4039 [Add forge now] Add feature description in features page
 Migratedgitlab-migrationT4007 [Add forge now] Create user guide
 Migratedgitlab-migrationT4037 [Add forge now] Create moderation guide
 Migratedgitlab-migrationT3629 doc: Add a "how to save a forge" as in how it's currently done
 Migratedgitlab-migrationT4329 Deploy add-forge-now in production

Event Timeline

zack triaged this task as Low priority.Feb 21 2019, 8:20 PM
zack created this task.
zack added a subtask: T1537: semi-automated addition of new "forges".
olasd mentioned this in T1870: revamp archive coverage page to list instances of mentioned listers.Jul 1 2019, 6:31 PM
vlorentz added a parent task: T2199: "Save Forge Now".Jan 22 2020, 4:42 PM
rdicosmo raised the priority of this task from Low to Normal.Mar 4 2021, 10:21 AM
rdicosmo merged a task: T2199: "Save Forge Now".
rdicosmo added a subscriber: douardda.
rdicosmo added a subscriber: rdicosmo.
rdicosmo added a project: Roadmap 2021.Mar 8 2021, 9:43 AM
rdicosmo updated the task description. (Show Details)Mar 8 2021, 10:13 AM
rdicosmo added a project: meta-task.
zack renamed this task from save "forge" now to Save "forge" now.Mar 31 2021, 11:07 AM
rdicosmo updated the task description. (Show Details)Mar 31 2021, 12:35 PM
vlorentz assigned this task to ardumont.Apr 23 2021, 4:48 PM
ardumont added a subtask: T3629: doc: Add a "how to save a forge" as in how it's currently done.Oct 6 2021, 6:24 PM
ardumont closed subtask T3629: doc: Add a "how to save a forge" as in how it's currently done as Resolved.Oct 8 2021, 5:55 PM
ardumont updated the task description. (Show Details)Oct 19 2021, 2:01 PM
ardumont renamed this task from Save "forge" now to Add "forge" now.Nov 8 2021, 11:23 AM
moranegg added a subscriber: moranegg.Nov 8 2021, 10:34 PM

This is an example of a use-case in Open Science:
An institution has a forge and wishes to add the forge as a whole to SWH.

Is the request for regular pulling?
If yes, it should be added for visibility on the home page of the archive.
If no, it should be clear this is a one shot thing.

zack added a comment.Nov 9 2021, 9:48 AM
In T1538#73659, @moranegg wrote:

Is the request for regular pulling?
If yes, it should be added for visibility on the home page of the archive.
If no, it should be clear this is a one shot thing.

Thanks a lot for this use case.

About the last point, do we really want to support archiving a forge only once?
In my mind, it has always been a first step when archiving a new forge, which we will then crawl regularly forever (until it shuts down, which we should be able to detect automatically eventually).
Other than that, I don't think it is in archival interest to not also regularly craw a forge that we know exists, after having archived it once.

ardumont added a comment.Nov 9 2021, 10:00 AM

Is the request for regular pulling?

It is.

The requisite is that first listing should happen quickly and then, regular "re-listing"
(and crawling) happens the same way the remaining part of the stack does (see scheduler
heuristics).

ardumont mentioned this in T2879: Finalize nixguix loader implementation.Jan 7 2022, 12:18 PM
bchauvet added a subtask: T4047: User interface to submit and follow Add forge now requests.Mar 16 2022, 11:32 AM
bchauvet added a subtask: T4058: Add forge now request processing admin tools.Mar 17 2022, 4:38 PM
bchauvet removed a subtask: T1537: semi-automated addition of new "forges".Mar 17 2022, 4:41 PM
bchauvet added a subtask: T4059: Add forge now documentation.Mar 17 2022, 4:44 PM
bchauvet removed a subtask: T3629: doc: Add a "how to save a forge" as in how it's currently done.Mar 17 2022, 4:47 PM
bchauvet removed a subtask: T3733: Develop a "Add forge now" mock-up.
bchauvet added a project: Roadmap 2022.Mar 23 2022, 4:36 PM
ardumont edited projects, added Add Forge Now ; removed Web app.Mar 24 2022, 11:01 AM
bchauvet raised the priority of this task from Normal to High.Mar 25 2022, 5:30 PM
bchauvet moved this task from Backlog to Work in progress on the Roadmap 2022 board.Mar 31 2022, 4:16 PM
ardumont added a subtask: T4329: Deploy add-forge-now in production.Jun 14 2022, 9:34 AM
ardumont changed the status of subtask T4329: Deploy add-forge-now in production from Open to Work in Progress.
ardumont closed subtask T4329: Deploy add-forge-now in production as Resolved.Jun 22 2022, 12:52 PM
gitlab-migration changed the status of subtask T4329: Deploy add-forge-now in production from Resolved to Migrated.Oct 19 2022, 6:07 PM
gitlab-migration closed subtask T4059: Add forge now documentation as Migrated.Jan 8 2023, 5:03 PM
gitlab-migration closed this task as Migrated.Jan 8 2023, 10:21 PM
gitlab-migration claimed this task.
gitlab-migration added subscribers: ardumont, gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration closed subtask T4047: User interface to submit and follow Add forge now requests as Migrated.Jan 8 2023, 10:24 PM
gitlab-migration closed subtask T4058: Add forge now request processing admin tools as Migrated.
About · Developer information · Legal