Differential D8663 Diff 31926 swh/lister/hackage/__init__.py

Changeset View

Standalone View

swh/lister/hackage/init.py

Show All 14 Lines
and build origins urls.		and build origins urls.

As of August 2022 `hackage.haskell.org`_ list 15536 package names.		As of August 2022 `hackage.haskell.org`_ list 15536 package names.

Origins retrieving strategy		Origins retrieving strategy
---------------------------		---------------------------

To get a list of all package names we make a POST call to		To get a list of all package names we make a POST call to
`https://hackage.haskell.org/packages/search` endpoint with some params given as		``https://hackage.haskell.org/packages/search`` endpoint with some params given as
json data.		json data.

Default params::		Default params::

{		{
"page": 0,		"page": 0,
"sortColumn": "default",		"sortColumn": "default",
"sortDirection": "ascending",		"sortDirection": "ascending",
"searchQuery": "(deprecated:any)",		"searchQuery": "(deprecated:any)",
}		}

The page size is 50. The lister will make has much http api call has needed to get		The page size is 50. The lister will make has much http api call has needed to get
all results.		all results.

		For incremental mode we expand the search query with ``lastUpload`` greater than
		``state.last_listing_date``, the api will return all new or updated package names since
		vlorentzUnsubmitted Done Inline Actions double backticks for monospace vlorentz: double backticks for monospace
		last run.

Page listing		Page listing
------------		------------

The result is paginated, each page is 50 records long.		The result is paginated, each page is 50 records long.

Entry data set example::		Entry data set example::

{		{
Show All 9 Lines	Entry data set example::
],		],
"votes": 1.5,		"votes": 1.5,
}		}

Origins from page		Origins from page
-----------------		-----------------

The lister yields 50 origins url per page.		The lister yields 50 origins url per page.
Each ListedOrigin has a `last_update` date set.		Each ListedOrigin has a ``last_update`` date set.

Running tests		Running tests
-------------		-------------

Activate the virtualenv and run from within swh-lister directory::		Activate the virtualenv and run from within swh-lister directory::

pytest -s -vv --log-cli-level=DEBUG swh/lister/hackage/tests		pytest -s -vv --log-cli-level=DEBUG swh/lister/hackage/tests

Show All 28 Lines