Page MenuHomeSoftware Heritage
Create Task
Maniphest T4233

Ingest Arch Linux
Closed, MigratedEdits Locked
Actions

Description

  • D7812: Lister draft (not landed)
  • D7812#205246: Discussion about current caveats and possible improvements based on ^
  • D7894, D8259, D8339: New lister implementation
  • D7894#207675, T4233#89838: Lister run ok within docker
  • D7995, D8264: Loader
  • Loader run ok within docker
  • D8259: Document Lister
  • D7995: Document Loader
  • Deploy to staging
  • Papercuts fixing
  • Call for public review on results ^
  • Deploy in production when green light

Revisions and Commits

rDLDBASE Generic VCS/Package Loader
ClosedD8264 Arch: Adapt the loader to manage 'arch_metadata' from 'extra_loader_arguments'
AbandonedD8554 arch/aur: Align loader parameters with lister output
D7995rDLDBASEb6af2638c11a Arch Linux loader
rDLS Listers
ClosedD8259 arch: Extra_loader_arguments consistency
ClosedD8339 Arch: Add throttling retry for scrapping and resources download
AbandonedD8553 arch/aur: Use dicts instead of lists as extra loader arguments
AbandonedD7812 [WIP] Add arch lister module.
D7894rDLS1bf11aa26d92 Add arch lister module (origins from archives).

Related Objects
Search...

StatusAssignedTask
Unknown Object (Maniphest Task)
 Migratedgitlab-migrationT4233 Ingest Arch Linux

Event Timeline

ardumont triaged this task as Normal priority.May 11 2022, 3:17 PM
ardumont created this task.
ardumont created this object in space Restricted Space.
ardumont added a revision: D7812: [WIP] Add arch lister module..May 11 2022, 3:26 PM
ardumont shifted this object from the Restricted Space space to the S1 Public space.
ardumont added a subscriber: franckbret.
franckbret mentioned this in D7894: Add arch lister module (origins from archives)..May 25 2022, 3:08 PM
franckbret mentioned this in D7897: [WIP] Add arch lister module (origins from archives)..May 25 2022, 3:31 PM
ardumont updated the task description. (Show Details)May 25 2022, 3:32 PM
ardumont added a revision: D7894: Add arch lister module (origins from archives)..
ardumont updated the task description. (Show Details)Jun 15 2022, 10:47 AM
franckbret added a revision: D7995: Arch Linux loader.Jun 16 2022, 9:38 AM
ardumont updated the task description. (Show Details)Jun 16 2022, 10:02 AM
ardumont updated the task description. (Show Details)Jun 16 2022, 2:47 PM
franckbret added a commit: rDLS1bf11aa26d92: Add arch lister module (origins from archives)..Jun 17 2022, 9:23 AM
franckbret added a commit: rDLDBASEb6af2638c11a: Arch Linux loader.Jun 17 2022, 9:33 AM
ardumont updated the task description. (Show Details)Jun 17 2022, 9:53 AM
bchauvet moved this task from Restricted Project Column to Restricted Project Column on the Unknown Object (Project) board.Jun 21 2022, 2:37 PM
franckbret mentioned this in D8051: Arch, use **kwargs on task initialisation instead of named args..Jun 29 2022, 3:56 PM
franckbret mentioned this in rDLDBASE01547b8edff5: Arch, use **kwargs on task initialisation instead of named args..Jun 29 2022, 5:20 PM
bchauvet removed a project: Archive coverage.Aug 18 2022, 2:16 PM
bchauvet moved this task from Restricted Project Column to Restricted Project Column on the Unknown Object (Project) board.
bchauvet moved this task from Restricted Project Column to Restricted Project Column on the Unknown Object (Project) board.
bchauvet edited projects, added Archive coverage; removed Unknown Object (Project).
franckbret mentioned this in D8259: arch: Extra_loader_arguments consistency.Aug 19 2022, 6:38 AM
franckbret mentioned this in D8264: Arch: Adapt the loader to manage 'arch_metadata' from 'extra_loader_arguments'.Aug 19 2022, 12:06 PM
franckbret mentioned this in rDLS7dd412e5538d: arch: Extra_loader_arguments consistency + documentation.Aug 19 2022, 3:53 PM
franckbret mentioned this in rDLDBASE9f8bdceed14a: Arch: Adapt the loader to manage 'arch_metadata' from 'extra_loader_arguments'.
franckbret added a comment.Aug 26 2022, 5:11 PM

Arch Linux Lister Docker Report

The lister takes a lot of time and fail on max retries when scraping repository directory (It has run fine a few weeks ago.). Not sure at this point, but I suspect that's a random problem related to network / http server. WIll run it multiple time to see if it failed on the same resource.

By the way I guess that the we need to define a strategy for those exceptions.

franckbret updated the task description. (Show Details)Aug 26 2022, 5:20 PM
franckbret updated the task description. (Show Details)
franckbret updated the task description. (Show Details)Aug 30 2022, 10:06 AM
franckbret updated the task description. (Show Details)Aug 30 2022, 10:14 AM
franckbret added a comment.Aug 30 2022, 10:28 AM
In T4233#89838, @franckbret wrote:

Arch Linux Lister Docker Report

The lister takes a lot of time and fail on max retries when scraping repository directory (It has run fine a few weeks ago.). Not sure at this point, but I suspect that's a random problem related to network / http server. WIll run it multiple time to see if it failed on the same resource.

By the way I guess that the we need to define a strategy for those exceptions.

Looks like D8339 helps the lister complete successfully

Task swh.lister.arch.tasks.ArchListerTask[a27f6656-936b-40c1-a1de-23c626d9a7e3] succeeded in 2188.289039958996s: {'pages': 9, 'origins': 36638}
franckbret mentioned this in D8339: Arch: Add throttling retry for scrapping and resources download.Aug 30 2022, 1:12 PM
anlambert added projects: Arch Lister, Arch loader.Sep 6 2022, 3:12 PM
anlambert added a revision: D8553: arch/aur: Use dicts instead of lists as extra loader arguments.Sep 27 2022, 5:16 PM
anlambert added a revision: D8554: arch/aur: Align loader parameters with lister output.
bchauvet added a revision: D8339: Arch: Add throttling retry for scrapping and resources download.Sep 30 2022, 10:09 AM
bchauvet added a revision: D8264: Arch: Adapt the loader to manage 'arch_metadata' from 'extra_loader_arguments'.Sep 30 2022, 10:16 AM
bchauvet added a revision: D8259: arch: Extra_loader_arguments consistency.Sep 30 2022, 10:20 AM
gitlab-migration closed this task as Migrated.Jan 8 2023, 10:24 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal