diff --git a/PKG-INFO b/PKG-INFO index de259a8..c4661ac 100644 --- a/PKG-INFO +++ b/PKG-INFO @@ -1,32 +1,32 @@ Metadata-Version: 2.1 Name: swh.loader.bzr -Version: 1.3.0 +Version: 1.3.1 Summary: Software Heritage Bazaar/Breezy intent Home-page: https://forge.softwareheritage.org/diffusion/DLDBZR/ Author: Software Heritage developers Author-email: swh-devel@inria.fr License: UNKNOWN Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest Project-URL: Funding, https://www.softwareheritage.org/donate Project-URL: Source, https://forge.softwareheritage.org/source/swh-loader-bzr Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-loader-bzr/ Platform: UNKNOWN Classifier: Programming Language :: Python :: 3 Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Classifier: Operating System :: OS Independent Classifier: Development Status :: 3 - Alpha Requires-Python: >=3.7 Description-Content-Type: text/x-rst Provides-Extra: testing License-File: LICENSE License-File: AUTHORS Software Heritage - Bazaar/Breezy loader ======================================== Loader for `Bazaar `_ and `Breezy `_ repositories. Breezy is a friendly fork of Bazaar that supports the Bazaar file format and network protocol. diff --git a/debian/changelog b/debian/changelog index 6f50767..7c47915 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,76 +1,78 @@ -swh-loader-bzr (1.3.0-1~swh1~bpo10+1) buster-swh; urgency=medium +swh-loader-bzr (1.3.1-1~swh1) unstable-swh; urgency=medium - * Rebuild for buster-swh + * New upstream release 1.3.1 - (tagged by Antoine Lambert + on 2022-04-29 12:11:19 +0200) + * Upstream changes: - version 1.3.1 - -- Software Heritage autobuilder (on jenkins-debian1) Mon, 25 Apr 2022 09:56:38 +0000 + -- Software Heritage autobuilder (on jenkins-debian1) Fri, 29 Apr 2022 10:17:53 +0000 swh-loader-bzr (1.3.0-1~swh1) unstable-swh; urgency=medium * New upstream release 1.3.0 - (tagged by Valentin Lorentz on 2022-04-25 11:48:10 +0200) * Upstream changes: - v1.3.0 - * Update for swh.loader.core 3.0.0 - * pre-commit and pytest maintenance -- Software Heritage autobuilder (on jenkins-debian1) Mon, 25 Apr 2022 09:54:58 +0000 swh-loader-bzr (1.2.0-1~swh1) unstable-swh; urgency=medium * New upstream release 1.2.0 - (tagged by Antoine R. Dumont (@ardumont) on 2022-02-23 09:55:46 +0100) * Upstream changes: - v1.2.0 - Support new repositories with old branches - loader: Upgrade repository with old formats prior to loading -- Software Heritage autobuilder (on jenkins-debian1) Wed, 23 Feb 2022 09:01:43 +0000 swh-loader-bzr (1.1.0-1~swh3) unstable-swh; urgency=medium * Bump new release -- Antoine R. Dumont (@ardumont) Fri, 11 Feb 2022 17:45:15 +0100 swh-loader-bzr (1.1.0-1~swh2) unstable-swh; urgency=medium * Bump new release -- Antoine R. Dumont (@ardumont) Fri, 11 Feb 2022 17:34:54 +0100 swh-loader-bzr (1.1.0-1~swh1) unstable-swh; urgency=medium * New upstream release 1.1.0 - (tagged by Antoine R. Dumont (@ardumont) on 2022-02-10 16:55:44 +0100) * Upstream changes: - v1.1.0 - Add fast-path to the bzr loader - Add type annotations, and fix issues found by mypy - pre- commit: Bump hooks and add new one to check commit message spelling -- Software Heritage autobuilder (on jenkins-debian1) Thu, 10 Feb 2022 16:40:56 +0000 swh-loader-bzr (1.0.1-1~swh1) unstable-swh; urgency=medium * New upstream release 1.0.1 - (tagged by Antoine R. Dumont (@ardumont) on 2022-02-09 16:14:32 +0100) * Upstream changes: - v1.0.1 - tests: Reference tasks module for tests to be ok with debian build - MANIFEST.in: Reference the tests data folder -- Software Heritage autobuilder (on jenkins-debian1) Wed, 09 Feb 2022 15:20:33 +0000 swh-loader-bzr (1.0.0-1~swh1) unstable-swh; urgency=medium * New upstream release 1.0.0 - (tagged by Antoine R. Dumont (@ardumont) on 2022-02-09 15:31:34 +0100) * Upstream changes: - v1.0.0 - Add incremental support for the bzr loader - Add bzr loader task - Use `branch` instead of `clone` - requirements-test: Pin pytest to < 7.0.0 -- Software Heritage autobuilder (on jenkins-debian1) Wed, 09 Feb 2022 14:39:13 +0000 swh-loader-bzr (0.1.0-1~swh1) unstable-swh; urgency=medium * Initial release -- Antoine R. Dumont (@ardumont) Wed, 09 Feb 2022 10:10:52 +0100 diff --git a/docs/how-bzr-works.rst b/docs/how-bzr-works.rst index d7c5839..4c13a04 100644 --- a/docs/how-bzr-works.rst +++ b/docs/how-bzr-works.rst @@ -1,41 +1,39 @@ .. _how-bzr-works: Software Heritage - How Bazaar/Breezy works =========================================== In Bazaar, a repository is simply the store of revisions. It's a storing backend and does not have to carry any semantic purpose for the project(s) it's holding. What users are really dealing with are branches. -A branch is an ordered set of revisions that describes the history of a set of files. It -corresponds to a folder on the file system, and can only have a single head: if two -clones of a branch diverge, the only way of uniting them is by merging one into the -other. A branch needs to have a repository to store its revisions, but multiple branches -can share the same repository. +A branch is an ordered set of revisions that describes the history of a set of files. +Like in Git, it's a pointer to a single revision. It corresponds to a folder on the +file system, and can only have a single head: if two clones of a branch diverge, +the only way of uniting them is by merging one into the other. A branch needs to +have a repository to store its revisions, but multiple branches can share the same repository. + +Note: there isn't a "Breezy" format, just the different Bazaar formats which are supported by Breezy, along with e.g. the Git format. Bazaar does not have a very strong opinion on how it should be used and supports multiple different workflows, even a centralized one with bound branches. We need to pick the most "workflow-agnostic" way of saving Bazaar repositories... or rather branches. For our purposes, we will treat each branch as a separate origin, since we have no way of knowing if branches inside a repository are related in bzr terms, and also because we de-duplicate on the SWH side: - From a user standpoint, they will most likely be searching by branch. If they search by shared repository, they will search with a prefix of a branch, which should also work - Since bzr branches do *not* have multiple heads, we don't have to worry about any sort of mapping, we will simply have HEAD - Tags are per-branch, so that also works - Ghost revisions can be saved even if we don't have the target revision since that's how the `nixguix` loader does it Not resolved yet: - - We have to look for stacked branches, how they work and if we can do anything more - interesting than just failing. - - Bazaar is able to store empty directories, does SWH handle them? - - What do we do about multiple authors (they are line separated) in each commit? - - What do we do about bug fixes metadata in each commit? - - What do we do about branch config? + - Bazaar is able to store empty directories, does SWH handle them? (T4201) + - What do we do about multiple authors (they are line separated) in each commit? (T3887) diff --git a/swh.loader.bzr.egg-info/PKG-INFO b/swh.loader.bzr.egg-info/PKG-INFO index de259a8..c4661ac 100644 --- a/swh.loader.bzr.egg-info/PKG-INFO +++ b/swh.loader.bzr.egg-info/PKG-INFO @@ -1,32 +1,32 @@ Metadata-Version: 2.1 Name: swh.loader.bzr -Version: 1.3.0 +Version: 1.3.1 Summary: Software Heritage Bazaar/Breezy intent Home-page: https://forge.softwareheritage.org/diffusion/DLDBZR/ Author: Software Heritage developers Author-email: swh-devel@inria.fr License: UNKNOWN Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest Project-URL: Funding, https://www.softwareheritage.org/donate Project-URL: Source, https://forge.softwareheritage.org/source/swh-loader-bzr Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-loader-bzr/ Platform: UNKNOWN Classifier: Programming Language :: Python :: 3 Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Classifier: Operating System :: OS Independent Classifier: Development Status :: 3 - Alpha Requires-Python: >=3.7 Description-Content-Type: text/x-rst Provides-Extra: testing License-File: LICENSE License-File: AUTHORS Software Heritage - Bazaar/Breezy loader ======================================== Loader for `Bazaar `_ and `Breezy `_ repositories. Breezy is a friendly fork of Bazaar that supports the Bazaar file format and network protocol. diff --git a/swh/loader/bzr/tasks.py b/swh/loader/bzr/tasks.py index d192163..964020e 100644 --- a/swh/loader/bzr/tasks.py +++ b/swh/loader/bzr/tasks.py @@ -1,26 +1,26 @@ # Copyright (C) 2022 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU General Public License version 3, or any later version # See top-level LICENSE file for more information -from typing import Optional - from celery import shared_task from swh.loader.core.utils import parse_visit_date from .loader import BazaarLoader +def _process_kwargs(kwargs): + if "visit_date" in kwargs: + kwargs["visit_date"] = parse_visit_date(kwargs["visit_date"]) + return kwargs + + @shared_task(name=__name__ + ".LoadBazaar") -def load_bzr( - *, url: str, directory: Optional[str] = None, visit_date: Optional[str] = None -): +def load_bzr(**kwargs): """Bazaar repository loading Args: see :func:`BazaarLoader` constructor. """ - loader = BazaarLoader.from_configfile( - url=url, directory=directory, visit_date=parse_visit_date(visit_date) - ) + loader = BazaarLoader.from_configfile(**_process_kwargs(kwargs)) return loader.load() diff --git a/swh/loader/bzr/tests/test_tasks.py b/swh/loader/bzr/tests/test_tasks.py index 7e7158d..1d6c7f6 100644 --- a/swh/loader/bzr/tests/test_tasks.py +++ b/swh/loader/bzr/tests/test_tasks.py @@ -1,27 +1,76 @@ # Copyright (C) 2022 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU General Public License version 3, or any later version # See top-level LICENSE file for more information +import uuid + +import pytest + +from swh.scheduler.model import ListedOrigin, Lister +from swh.scheduler.utils import create_origin_task_dict + + +@pytest.fixture(autouse=True) +def celery_worker_and_swh_config(swh_scheduler_celery_worker, swh_config): + pass + + +@pytest.fixture +def bzr_lister(): + return Lister(name="bzr-lister", instance_name="example", id=uuid.uuid4()) + + +@pytest.fixture +def bzr_listed_origin(bzr_lister): + return ListedOrigin( + lister_id=bzr_lister.id, url="https://bzr.example.org/repo", visit_type="bzr" + ) + def test_loader( - mocker, swh_config, swh_scheduler_celery_app, swh_scheduler_celery_worker + mocker, + swh_scheduler_celery_app, ): mock_loader = mocker.patch("swh.loader.bzr.loader.BazaarLoader.load") mock_loader.return_value = {"status": "eventful"} res = swh_scheduler_celery_app.send_task( "swh.loader.bzr.tasks.LoadBazaar", kwargs={ "url": "origin_url", "directory": "/some/repo", "visit_date": "now", }, ) assert res res.wait() assert res.successful() assert res.result == {"status": "eventful"} mock_loader.assert_called_once_with() + + +def test_loader_for_listed_origin( + mocker, + swh_scheduler_celery_app, + bzr_lister, + bzr_listed_origin, +): + mock_loader = mocker.patch("swh.loader.bzr.loader.BazaarLoader.load") + mock_loader.return_value = {"status": "eventful"} + + task_dict = create_origin_task_dict(bzr_listed_origin, bzr_lister) + + res = swh_scheduler_celery_app.send_task( + "swh.loader.bzr.tasks.LoadBazaar", + kwargs=task_dict["arguments"]["kwargs"], + ) + + assert res + res.wait() + assert res.successful() + + assert res.result == {"status": "eventful"} + mock_loader.assert_called_once_with() diff --git a/tox.ini b/tox.ini index 7882424..be4644e 100644 --- a/tox.ini +++ b/tox.ini @@ -1,74 +1,74 @@ [tox] envlist=black,flake8,mypy,py3 [testenv] extras = testing deps = pytest-cov commands = pytest --doctest-modules \ {envsitepackagesdir}/swh/loader/bzr \ --cov={envsitepackagesdir}/swh/loader/bzr \ --cov-branch {posargs} [testenv:black] skip_install = true deps = black==22.3.0 commands = {envpython} -m black --check swh [testenv:flake8] skip_install = true deps = flake8==4.0.1 flake8-bugbear==22.3.23 commands = {envpython} -m flake8 [testenv:mypy] extras = testing deps = - mypy==0.920 + mypy==0.942 commands = mypy swh # build documentation outside swh-environment using the current # git HEAD of swh-docs, is executed on CI for each diff to prevent # breaking doc build [testenv:sphinx] whitelist_externals = make usedevelop = true extras = testing deps = # fetch and install swh-docs in develop mode -e git+https://forge.softwareheritage.org/source/swh-docs#egg=swh.docs setenv = SWH_PACKAGE_DOC_TOX_BUILD = 1 # turn warnings into errors SPHINXOPTS = -W commands = make -I ../.tox/sphinx/src/swh-docs/swh/ -C docs # build documentation only inside swh-environment using local state # of swh-docs package [testenv:sphinx-dev] whitelist_externals = make usedevelop = true extras = testing deps = # install swh-docs in develop mode -e ../swh-docs setenv = SWH_PACKAGE_DOC_TOX_BUILD = 1 # turn warnings into errors SPHINXOPTS = -W commands = make -I ../.tox/sphinx-dev/src/swh-docs/swh/ -C docs