- D8287: Implement lister
- D8305, D8321, D8330: Implement loader
- T4465#89806: Lister run in docker
- T4465#89812: Loader run in docker
- D8287: Document lister
- D8305: Document loader
- T4465: Deploy on staging
- Call for public review
- Deploy on production
Description
Description
Revisions and Commits
Revisions and Commits
Status | Assigned | Task | ||
---|---|---|---|---|
Unknown Object (Maniphest Task) | ||||
Migrated | gitlab-migration | T4465 Ingest pub.dev (Dart, Flutter) | ||
Migrated | gitlab-migration | T4517 staging: Deploy pubdev | ||
Migrated | gitlab-migration | T4628 Pubdev: load authors from API instead of pubspec.yaml | ||
Migrated | gitlab-migration | T4665 [pudbev lister] use alternative packages list endpoint |
Event Timeline
Comment Actions
Lister runs on Docker report
Lister runs fine on Docker and found 33564 origins.
It takes less than 20s to complete, it runs fast and without errors.
Lister succeeded in 14.662720984983025s: {'pages': 1, 'origins': 33564} swh-scheduler=# select count(*) from listed_origins where visit_type = 'pubdev'; count ------- 33564
Comment Actions
Loader runs on Docker report
Loader runs fine on docker.
I've first launched 100 and then 1900 loader tasks, it complete in less than an hour.
swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='successful'; count ------- 1450 swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='failed'; count ------- 550
Comment Actions
D8330 fix the large amount of failed visits. I've launched 10000 tasks with following results:
swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='successful'; count ------- 9977 (1 row) swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='failed'; count ------- 23 (1 row)