Page MenuHomeSoftware Heritage
Create Task
Maniphest T1380

npm lister
Closed, MigratedEdits Locked
Actions

Description

We want to be able to list all available packages on npm in order to load their content into the archive.

Revisions and Commits

rDLS Listers
D706rDLS605a67f51d7d swh.lister.npm : Add a lister of all available packages in the npm registry

Related Objects
Search...

Event Timeline

anlambert triaged this task as Normal priority.Nov 22 2018, 3:53 PM
anlambert created this task.
anlambert added a comment.EditedNov 22 2018, 4:00 PM

The npm registry is a CouchDB database located at https://replicate.npmjs.com.

The following endpoint enables to list all registered packages: https://replicate.npmjs.com/_all_docs?limit=100

We should be able to use the SWHIndexingHttpLister [1] using the recommended CouchDB pagination method [2]

[1] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/core/indexing_lister.py

[2] http://docs.couchdb.org/en/stable/ddocs/views/pagination.html#paging-alternate-method

anlambert claimed this task.Nov 23 2018, 4:34 PM
anlambert updated the task description. (Show Details)
anlambert added a revision: D706: Add a lister of all available packages in the npm registry.Nov 23 2018, 4:46 PM
anlambert closed this task as Resolved by committing rDLS605a67f51d7d: swh.lister.npm : Add a lister of all available packages in the npm registry.Nov 26 2018, 11:05 AM
anlambert added a commit: rDLS605a67f51d7d: swh.lister.npm : Add a lister of all available packages in the npm registry.
anlambert mentioned this in T1398: npm incremental lister.Nov 29 2018, 8:09 PM
anlambert added a project: Npm Lister.Nov 30 2018, 7:37 PM
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 4:26 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal