Page MenuHomeSoftware Heritage

npm lister
Closed, ResolvedPublic

Description

We want to be able to list all available packages on npm in order to load their content into the archive.

Event Timeline

anlambert triaged this task as Normal priority.Nov 22 2018, 3:53 PM
anlambert created this task.

The npm registry is a CouchDB database located at https://replicate.npmjs.com.

The following endpoint enables to list all registered packages: https://replicate.npmjs.com/_all_docs?limit=100

We should be able to use the SWHIndexingHttpLister [1] using the recommended CouchDB pagination method [2]

[1] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/core/indexing_lister.py

[2] http://docs.couchdb.org/en/stable/ddocs/views/pagination.html#paging-alternate-method