Page MenuHomeSoftware Heritage

Ingest crates.io (Rust)
Open, NormalPublic

Description

  • T1424, D7367: Implement lister
  • D7501: Implement loader
  • T4104#87630: Run in docker
  • D8199: Document lister
  • Document loader
  • Deploy on staging
  • Call for public review
  • Deploy on production

Event Timeline

ardumont updated the task description. (Show Details)
bchauvet mentioned this in Unknown Object (Maniphest Task).May 12 2022, 10:27 AM
bchauvet added a parent task: Unknown Object (Maniphest Task).May 12 2022, 10:32 AM

Hello,
The crates lister (stateless) and loader have landed.
I just solved some discovered issues while running lister and loader on the Docker env ( D8049 ).

The lister and the loader runs fine through cli and scheduler. As of today the lister get 86979 crates origins in less than 5 minutes.
The loader runs fine too through the scheduler.

What the next steps here? Do you want to move to production or does it need to be incremental first?

ardumont updated the task description. (Show Details)

What the next steps here?

As per the plan in the description, trigger a run in our staging infra (production-like
but not production yet). After that, we'll do a public call for review on the swh-devel
mailing list. And if all is fine, that can run in production.

Do you want to move to production or does it need to be incremental first?

I guess you can start working on making the listing incremental if you see a way to do
it already.


Is the documentation for that new lister/loader documented already? (I do not recall).

Cheers,

Ok,

Will work on the incremental lister, and then document (not already done).