Page MenuHomeSoftware Heritage

Ingest pub.dev (Dart, Flutter)
Closed, MigratedEdits Locked

Description

Event Timeline

Lister runs on Docker report

Lister runs fine on Docker and found 33564 origins.
It takes less than 20s to complete, it runs fast and without errors.

Lister succeeded in 14.662720984983025s: {'pages': 1, 'origins': 33564}

swh-scheduler=# select count(*) from listed_origins where visit_type = 'pubdev';
 count 
-------
 33564

Loader runs on Docker report

Loader runs fine on docker.
I've first launched 100 and then 1900 loader tasks, it complete in less than an hour.

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='successful';
 count 
-------
  1450

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='failed';
 count 
-------
   550
franckbret updated the task description. (Show Details)
franckbret updated the task description. (Show Details)
vlorentz triaged this task as Normal priority.Aug 26 2022, 5:36 PM

Loader runs on Docker report

Loader runs fine on docker.
I've first launched 100 and then 1900 loader tasks, it complete in less than an hour.

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='successful';
 count 
-------
  1450

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='failed';
 count 
-------
   550

D8330 fix the large amount of failed visits. I've launched 10000 tasks with following results:

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='successful';
 count 
-------
  9977
(1 row)

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='pubdev' and last_visit_status='failed';                                                       
 count 
-------
    23
(1 row)
bchauvet added a parent task: Unknown Object (Maniphest Task).Sep 2 2022, 10:45 AM
ardumont changed the status of subtask T4517: staging: Deploy pubdev from Open to Work in Progress.Sep 9 2022, 9:51 AM
bchauvet added a subscriber: bchauvet.

Public review successfully validated, ready for production :)