Ingest bower.io (Javascript package manager)
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	franckbret
	Aug 29 2022, 5:26 PM

Description

D8333, D8336: Implement lister
D8336: Implement loader (Bower package loader is not needed as origins url are Git repositories)
Lister run in docker
Loader run in docker
D8333: Document lister
D8336: Document loader (Bower package loader is not needed as origins url are Git repositories)
T4555: Deploy on staging
Call for public review
Deploy on production

Revisions and Commits

rDLS Listers
	Closed	D8333 Bower: List origins from registry.bower.io
	Closed	D8336 Bower: Set VISIT_TYPE as 'git'

Related Objects
Search...

Status	Assigned	Task
		Unknown Object (Maniphest Task)
Migrated	gitlab-migration	T4475 Ingest bower.io (Javascript package manager)
Migrated	gitlab-migration	T4555 staging: Deploy bower lister

Event Timeline

franckbret created this task.Aug 29 2022, 5:26 PM

franckbret created this object in space S1 Public.

Lister runs fine on Docker and takes 30sec to list 68864 origins

swh-lister_1                        | [2022-08-30 06:31:51,945: INFO/ForkPoolWorker-1] Fetching URL https://registry.bower.io/packages with params {}
swh-lister_1                        | [2022-08-30 06:32:20,070: INFO/ForkPoolWorker-1] Task swh.lister.bower.tasks.BowerListerTask[20414db4-682b-4b77-b8be-cac95ec3cf83] succeeded in 28.162076047010487s: {'pages': 1, 'origins': 68864}

franckbret updated the task description. (Show Details)Aug 30 2022, 8:43 AM

I've launched 100 git tasks after running the lister on Docker.
Introspecting logs, not_found are 404 for repositories that do not exists anymore or No valid credentials provided.
Those with status 'failed' are mainly time out.

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='git' and last_visit_status='successful';
-[ RECORD 1 ]
count | 68

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='git' and last_visit_status='failed';                                                              
-[ RECORD 1 ]
count | 4

swh-scheduler=# select count(*) from origin_visit_stats where visit_type='git' and last_visit_status='not_found';
-[ RECORD 1 ]
count | 28

franckbret updated the task description. (Show Details)Aug 30 2022, 8:54 AM

franckbret updated the task description. (Show Details)Aug 30 2022, 9:07 AM

ardumont triaged this task as Normal priority.Aug 30 2022, 11:10 AM

ardumont added a project: Archive coverage.

bchauvet added a parent task: Unknown Object (Maniphest Task).Sep 2 2022, 10:50 AM

anlambert added a project: Bower lister.Sep 6 2022, 4:17 PM

ardumont updated the task description. (Show Details)Sep 26 2022, 1:26 PM

ardumont updated the task description. (Show Details)

ardumont changed the status of subtask T4555: staging: Deploy bower lister from Open to Work in Progress.Sep 26 2022, 1:39 PM

ardumont closed subtask T4555: staging: Deploy bower lister as Resolved.Sep 28 2022, 7:19 PM

ardumont updated the task description. (Show Details)Sep 28 2022, 7:27 PM

bchauvet added a revision: D8336: Bower: Set VISIT_TYPE as 'git'.Sep 30 2022, 10:11 AM