No OneTemporary
Actions

Size

16 KB

Subscribers

None

View Options

	diff --git a/PKG-INFO b/PKG-INFO
	index b588fce..1612f4c 100644
	--- a/PKG-INFO
	+++ b/PKG-INFO
	@@ -1,125 +1,125 @@
	Metadata-Version: 2.1
	Name: swh.lister
	-Version: 1.3.2
	+Version: 1.3.3
	Summary: Software Heritage lister
	Home-page: https://forge.softwareheritage.org/diffusion/DLSGH/
	Author: Software Heritage developers
	Author-email: swh-devel@inria.fr
	License: UNKNOWN
	Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest
	Project-URL: Funding, https://www.softwareheritage.org/donate
	Project-URL: Source, https://forge.softwareheritage.org/source/swh-lister
	Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-lister/
	Platform: UNKNOWN
	Classifier: Programming Language :: Python :: 3
	Classifier: Intended Audience :: Developers
	Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
	Classifier: Operating System :: OS Independent
	Classifier: Development Status :: 5 - Production/Stable
	Requires-Python: >=3.7
	Description-Content-Type: text/markdown
	Provides-Extra: testing
	License-File: LICENSE

	swh-lister
	==========

	This component from the Software Heritage stack aims to produce listings
	of software origins and their urls hosted on various public developer platforms
	or package managers. As these operations are quite similar, it provides a set of
	Python modules abstracting common software origins listing behaviors.

	It also provides several lister implementations, contained in the
	following Python modules:

	- `swh.lister.bitbucket`
	- `swh.lister.cgit`
	- `swh.lister.cran`
	- `swh.lister.debian`
	- `swh.lister.gitea`
	- `swh.lister.github`
	- `swh.lister.gitlab`
	- `swh.lister.gnu`
	- `swh.lister.launchpad`
	- `swh.lister.npm`
	- `swh.lister.packagist`
	- `swh.lister.phabricator`
	- `swh.lister.pypi`

	Dependencies
	------------

	All required dependencies can be found in the `requirements*.txt` files located
	at the root of the repository.

	Local deployment
	----------------

	## lister configuration

	Each lister implemented so far by Software Heritage (`bitbucket`, `cgit`, `cran`, `debian`,
	`gitea`, `github`, `gitlab`, `gnu`, `launchpad`, `npm`, `packagist`, `phabricator`, `pypi`)
	must be configured by following the instructions below (please note that you have to replace
	`<lister_name>` by one of the lister name introduced above).

	### Preparation steps

	1. `mkdir ~/.config/swh/`
	2. create configuration file `~/.config/swh/listers.yml`

	### Configuration file sample

	Minimalistic configuration shared by all listers to add in file `~/.config/swh/listers.yml`:

	```lang=yml
	scheduler:
	cls: 'remote'
	args:
	url: 'http://localhost:5008/'

	credentials: {}
	```

	Note: This expects scheduler (5008) service to run locally

	## Executing a lister

	Once configured, a lister can be executed by using the `swh` CLI tool with the
	following options and commands:

	```
	$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister <lister_name> [lister_parameters]
	```

	Examples:

	```
	$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister bitbucket

	$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister cran

	$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister gitea url=https://codeberg.org/api/v1/

	$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister gitlab url=https://salsa.debian.org/api/v4/

	$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister npm

	$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister pypi
	```

	Licensing
	---------

	This program is free software: you can redistribute it and/or modify it under
	the terms of the GNU General Public License as published by the Free Software
	Foundation, either version 3 of the License, or (at your option) any later
	version.

	This program is distributed in the hope that it will be useful, but WITHOUT ANY
	WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
	PARTICULAR PURPOSE. See the GNU General Public License for more details.

	See top-level LICENSE file for the full text of the GNU General Public License
	along with this program.


	diff --git a/swh.lister.egg-info/PKG-INFO b/swh.lister.egg-info/PKG-INFO
	index b588fce..1612f4c 100644
	--- a/swh.lister.egg-info/PKG-INFO
	+++ b/swh.lister.egg-info/PKG-INFO
	@@ -1,125 +1,125 @@
	Metadata-Version: 2.1
	Name: swh.lister
	-Version: 1.3.2
	+Version: 1.3.3
	Summary: Software Heritage lister
	Home-page: https://forge.softwareheritage.org/diffusion/DLSGH/
	Author: Software Heritage developers
	Author-email: swh-devel@inria.fr
	License: UNKNOWN
	Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest
	Project-URL: Funding, https://www.softwareheritage.org/donate
	Project-URL: Source, https://forge.softwareheritage.org/source/swh-lister
	Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-lister/
	Platform: UNKNOWN
	Classifier: Programming Language :: Python :: 3
	Classifier: Intended Audience :: Developers
	Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
	Classifier: Operating System :: OS Independent
	Classifier: Development Status :: 5 - Production/Stable
	Requires-Python: >=3.7
	Description-Content-Type: text/markdown
	Provides-Extra: testing
	License-File: LICENSE

	swh-lister
	==========

	This component from the Software Heritage stack aims to produce listings
	of software origins and their urls hosted on various public developer platforms
	or package managers. As these operations are quite similar, it provides a set of
	Python modules abstracting common software origins listing behaviors.

	It also provides several lister implementations, contained in the
	following Python modules:

	- `swh.lister.bitbucket`
	- `swh.lister.cgit`
	- `swh.lister.cran`
	- `swh.lister.debian`
	- `swh.lister.gitea`
	- `swh.lister.github`
	- `swh.lister.gitlab`
	- `swh.lister.gnu`
	- `swh.lister.launchpad`
	- `swh.lister.npm`
	- `swh.lister.packagist`
	- `swh.lister.phabricator`
	- `swh.lister.pypi`

	Dependencies
	------------

	All required dependencies can be found in the `requirements*.txt` files located
	at the root of the repository.

	Local deployment
	----------------

	## lister configuration

	Each lister implemented so far by Software Heritage (`bitbucket`, `cgit`, `cran`, `debian`,
	`gitea`, `github`, `gitlab`, `gnu`, `launchpad`, `npm`, `packagist`, `phabricator`, `pypi`)
	must be configured by following the instructions below (please note that you have to replace
	`<lister_name>` by one of the lister name introduced above).

	### Preparation steps

	1. `mkdir ~/.config/swh/`
	2. create configuration file `~/.config/swh/listers.yml`

	### Configuration file sample

	Minimalistic configuration shared by all listers to add in file `~/.config/swh/listers.yml`:

	```lang=yml
	scheduler:
	cls: 'remote'
	args:
	url: 'http://localhost:5008/'

	credentials: {}
	```

	Note: This expects scheduler (5008) service to run locally

	## Executing a lister

	Once configured, a lister can be executed by using the `swh` CLI tool with the
	following options and commands:

	```
	$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister <lister_name> [lister_parameters]
	```

	Examples:

	```
	$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister bitbucket

	$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister cran

	$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister gitea url=https://codeberg.org/api/v1/

	$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister gitlab url=https://salsa.debian.org/api/v4/

	$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister npm

	$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister pypi
	```

	Licensing
	---------

	This program is free software: you can redistribute it and/or modify it under
	the terms of the GNU General Public License as published by the Free Software
	Foundation, either version 3 of the License, or (at your option) any later
	version.

	This program is distributed in the hope that it will be useful, but WITHOUT ANY
	WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
	PARTICULAR PURPOSE. See the GNU General Public License for more details.

	See top-level LICENSE file for the full text of the GNU General Public License
	along with this program.


	diff --git a/swh/lister/cgit/lister.py b/swh/lister/cgit/lister.py
	index 4f55e36..9a19cd5 100644
	--- a/swh/lister/cgit/lister.py
	+++ b/swh/lister/cgit/lister.py
	@@ -1,216 +1,213 @@
	# Copyright (C) 2019-2021 The Software Heritage developers
	# License: GNU General Public License version 3, or any later version
	# See top-level LICENSE file for more information

	from datetime import datetime, timezone
	import logging
	import re
	from typing import Any, Dict, Iterator, List, Optional
	from urllib.parse import urljoin, urlparse

	from bs4 import BeautifulSoup
	import requests
	from requests.exceptions import HTTPError

	from swh.lister import USER_AGENT
	from swh.lister.pattern import CredentialsType, StatelessLister
	from swh.scheduler.interface import SchedulerInterface
	from swh.scheduler.model import ListedOrigin

	logger = logging.getLogger(__name__)

	Repositories = List[Dict[str, Any]]


	class CGitLister(StatelessLister[Repositories]):
	"""Lister class for CGit repositories.

	This lister will retrieve the list of published git repositories by
	parsing the HTML page(s) of the index retrieved at `url`.

	The lister currently defines 2 listing behaviors:

	- If the `base_git_url` is provided, the listed origin urls are computed out of the
	base git url link and the one listed in the main listed page (resulting in less
	HTTP queries than the 2nd behavior below). This is expected to be the main
	deployed behavior.

	- Otherwise (with no `base_git_url`), for each found git repository listed, one
	extra HTTP query is made at the given url found in the main listing page to gather
	published "Clone" URLs to be used as origin URL for that git repo. If several
	"Clone" urls are provided, prefer the http/https one, if any, otherwise fallback
	to the first one.

	"""

	LISTER_NAME = "cgit"

	def __init__(
	self,
	scheduler: SchedulerInterface,
	url: str,
	instance: Optional[str] = None,
	credentials: Optional[CredentialsType] = None,
	base_git_url: Optional[str] = None,
	):
	"""Lister class for CGit repositories.

	Args:
	url: main URL of the CGit instance, i.e. url of the index
	of published git repositories on this instance.
	instance: Name of cgit instance. Defaults to url's hostname
	if unset.
	base_git_url: Optional base git url which allows the origin url
	computations.

	"""
	if not instance:
	instance = urlparse(url).hostname
	assert instance is not None # Make mypy happy

	super().__init__(
	scheduler=scheduler, url=url, instance=instance, credentials=credentials,
	)

	self.session = requests.Session()
	self.session.headers.update(
	{"Accept": "application/html", "User-Agent": USER_AGENT}
	)
	self.base_git_url = base_git_url

	def _get_and_parse(self, url: str) -> BeautifulSoup:
	"""Get the given url and parse the retrieved HTML using BeautifulSoup"""
	response = self.session.get(url)
	response.raise_for_status()
	return BeautifulSoup(response.text, features="html.parser")

	def get_pages(self) -> Iterator[Repositories]:
	"""Generate git 'project' URLs found on the current CGit server
	The last_update date is retrieved on the list of repo page to avoid
	to compute it on the repository details which only give a date per branch
	"""
	next_page: Optional[str] = self.url
	while next_page:
	bs_idx = self._get_and_parse(next_page)

	page_results = []

	for tr in bs_idx.find("div", {"class": "content"}).find_all(
	"tr", {"class": ""}
	):
	repository_link = tr.find("a")["href"]
	repo_url = None
	git_url = None

	base_url = urljoin(self.url, repository_link).strip("/")
	if self.base_git_url: # mapping provided
	# computing git url
	git_url = base_url.replace(self.url, self.base_git_url)
	else:
	# we compute the git detailed page url from which we will retrieve
	# the git url (cf. self.get_origins_from_page)
	repo_url = base_url

	span = tr.find("span", {"class": re.compile("age-")})
	- if span:
	- last_updated_date = span["title"]
	- else:
	- last_updated_date = None
	+ last_updated_date = span.get("title") if span else None

	page_results.append(
	{
	"url": repo_url,
	"git_url": git_url,
	"last_updated_date": last_updated_date,
	}
	)

	yield page_results

	try:
	pager = bs_idx.find("ul", {"class": "pager"})

	current_page = pager.find("a", {"class": "current"})
	if current_page:
	next_page = current_page.parent.next_sibling.a["href"]
	next_page = urljoin(self.url, next_page)
	except (AttributeError, KeyError):
	# no pager, or no next page
	next_page = None

	def get_origins_from_page(
	self, repositories: Repositories
	) -> Iterator[ListedOrigin]:
	"""Convert a page of cgit repositories into a list of ListedOrigins."""
	assert self.lister_obj.id is not None

	for repo in repositories:
	origin_url = repo["git_url"] or self._get_origin_from_repository_url(
	repo["url"]
	)
	if origin_url is None:
	continue

	yield ListedOrigin(
	lister_id=self.lister_obj.id,
	url=origin_url,
	visit_type="git",
	last_update=_parse_last_updated_date(repo),
	)

	def _get_origin_from_repository_url(self, repository_url: str) -> Optional[str]:
	"""Extract the git url from the repository page"""
	try:
	bs = self._get_and_parse(repository_url)
	except HTTPError as e:
	logger.warning(
	"Unexpected HTTP status code %s on %s",
	e.response.status_code,
	e.response.url,
	)
	return None

	# origin urls are listed on the repository page
	# TODO check if forcing https is better or not ?
	# <link rel='vcs-git' href='git://...' title='...'/>
	# <link rel='vcs-git' href='http://...' title='...'/>
	# <link rel='vcs-git' href='https://...' title='...'/>
	urls = [x["href"] for x in bs.find_all("a", {"rel": "vcs-git"})]

	if not urls:
	return None

	# look for the http/https url, if any, and use it as origin_url
	for url in urls:
	if urlparse(url).scheme in ("http", "https"):
	origin_url = url
	break
	else:
	# otherwise, choose the first one
	origin_url = urls[0]
	return origin_url


	def _parse_last_updated_date(repository: Dict[str, Any]) -> Optional[datetime]:
	"""Parse the last updated date"""
	date = repository.get("last_updated_date")
	if not date:
	return None

	parsed_date = None
	for date_format in ("%Y-%m-%d %H:%M:%S %z", "%Y-%m-%d %H:%M:%S (%Z)"):
	try:
	parsed_date = datetime.strptime(date, date_format)
	# force UTC to avoid naive datetime
	if not parsed_date.tzinfo:
	parsed_date = parsed_date.replace(tzinfo=timezone.utc)
	break
	except Exception:
	pass

	if not parsed_date:
	logger.warning(
	"Could not parse %s last_updated date: %s", repository["url"], date,
	)

	return parsed_date

File Metadata

Mime Type: text/x-diff
Expires: Jul 4 2025, 8:11 AM (10 w, 7 h ago)
Storage Engine: blob
Storage Format: Raw Data
Storage Handle: 3306961

No OneTemporaryActions

View Options

File Metadata

Event Timeline

No OneTemporary
Actions