Page MenuHomeSoftware Heritage

Ingest Arch Linux
Open, NormalPublic

Description

  • D7812: Lister draft (not landed)
  • D7812#205246: Discussion about current caveats and possible improvements based on ^
  • D7894, D8259, D8339: New lister implementation
  • D7894#207675, T4233#89838: Lister run ok within docker
  • D7995, D8264: Loader
  • Loader run ok within docker
  • D8259: Document Lister
  • D7995: Document Loader
  • Deploy to staging
  • Papercuts fixing
  • Call for public review on results ^
  • Deploy in production when green light

Event Timeline

ardumont triaged this task as Normal priority.May 11 2022, 3:17 PM
ardumont created this task.
ardumont created this object in space Restricted Space.
ardumont shifted this object from the Restricted Space space to the S1 Public space.
ardumont added a subscriber: franckbret.
bchauvet moved this task from Restricted Project Column to Restricted Project Column on the Unknown Object (Project) board.Jun 21 2022, 2:37 PM
bchauvet moved this task from Restricted Project Column to Restricted Project Column on the Unknown Object (Project) board.
bchauvet moved this task from Restricted Project Column to Restricted Project Column on the Unknown Object (Project) board.
bchauvet edited projects, added Archive coverage; removed Unknown Object (Project).

Arch Linux Lister Docker Report

The lister takes a lot of time and fail on max retries when scraping repository directory (It has run fine a few weeks ago.). Not sure at this point, but I suspect that's a random problem related to network / http server. WIll run it multiple time to see if it failed on the same resource.

By the way I guess that the we need to define a strategy for those exceptions.

Arch Linux Lister Docker Report

The lister takes a lot of time and fail on max retries when scraping repository directory (It has run fine a few weeks ago.). Not sure at this point, but I suspect that's a random problem related to network / http server. WIll run it multiple time to see if it failed on the same resource.

By the way I guess that the we need to define a strategy for those exceptions.

Looks like D8339 helps the lister complete successfully

Task swh.lister.arch.tasks.ArchListerTask[a27f6656-936b-40c1-a1de-23c626d9a7e3] succeeded in 2188.289039958996s: {'pages': 9, 'origins': 36638}