diff --git a/PKG-INFO b/PKG-INFO index 233b713..164e16d 100644 --- a/PKG-INFO +++ b/PKG-INFO @@ -1,61 +1,61 @@ Metadata-Version: 2.1 Name: swh.loader.svn -Version: 1.3.1 +Version: 1.3.2 Summary: Software Heritage Loader SVN Home-page: https://forge.softwareheritage.org/diffusion/DLDSVN Author: Software Heritage developers Author-email: swh-devel@inria.fr License: UNKNOWN Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest Project-URL: Funding, https://www.softwareheritage.org/donate Project-URL: Source, https://forge.softwareheritage.org/source/swh-loader-svn Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-loader-svn/ Platform: UNKNOWN Classifier: Programming Language :: Python :: 3 Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Classifier: Operating System :: OS Independent Classifier: Development Status :: 5 - Production/Stable Requires-Python: >=3.7 Description-Content-Type: text/markdown Provides-Extra: testing License-File: LICENSE License-File: AUTHORS swh-loader-svn ============== The Software Heritage SVN Loader is a tool and a library to walk a remote svn repository and inject into the SWH dataset all contained files that weren't known before. The main entry points are - :class:`swh.loader.svn.loader.SvnLoader` for the main svn loader which ingests content out of a remote svn repository - :class:`swh.loader.svn.loader.SvnLoaderFromDumpArchive` which mounts a repository out of a svn dump prior to ingest it. - :class:`swh.loader.svn.loader.SvnLoaderFromRemoteDump` which mounts a repository with svnrdump prior to ingest its content. # CLI run With the configuration: /tmp/loader_svn.yml: ``` storage: cls: remote args: url: http://localhost:5002/ ``` Run: ``` swh loader --config-file /tmp/loader_svn.yml \ run svn ``` diff --git a/debian/changelog b/debian/changelog index c92b398..d0ef67e 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,828 +1,830 @@ -swh-loader-svn (1.3.1-1~swh1~bpo10+1) buster-swh; urgency=medium +swh-loader-svn (1.3.2-1~swh1) unstable-swh; urgency=medium - * Rebuild for buster-swh + * New upstream release 1.3.2 - (tagged by Antoine Lambert + on 2022-04-29 12:06:51 +0200) + * Upstream changes: - version 1.3.2 - -- Software Heritage autobuilder (on jenkins-debian1) Thu, 21 Apr 2022 13:29:55 +0000 + -- Software Heritage autobuilder (on jenkins-debian1) Fri, 29 Apr 2022 10:11:07 +0000 swh-loader-svn (1.3.1-1~swh1) unstable-swh; urgency=medium * New upstream release 1.3.1 - (tagged by Antoine Lambert on 2022-04-21 15:24:00 +0200) * Upstream changes: - version 1.3.1 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 21 Apr 2022 13:27:38 +0000 swh-loader-svn (1.3.0-1~swh1) unstable-swh; urgency=medium * New upstream release 1.3.0 - (tagged by Antoine Lambert on 2022-03-24 13:30:17 +0100) * Upstream changes: - version 1.3.0 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 24 Mar 2022 12:34:14 +0000 swh-loader-svn (1.2.1-1~swh1) unstable-swh; urgency=medium * New upstream release 1.2.1 - (tagged by Antoine Lambert on 2022-03-09 13:57:18 +0100) * Upstream changes: - version 1.2.1 -- Software Heritage autobuilder (on jenkins-debian1) Wed, 09 Mar 2022 13:00:41 +0000 swh-loader-svn (1.2.0-1~swh2) unstable-swh; urgency=medium * Bump new release -- Antoine R. Dumont (@ardumont) Thu, 17 Feb 2022 12:46:20 +0100 swh-loader-svn (1.2.0-1~swh1) unstable-swh; urgency=medium * New upstream release 1.2.0 - (tagged by Antoine Lambert on 2022-02-11 17:19:45 +0100) * Upstream changes: - version 1.2.0 -- Software Heritage autobuilder (on jenkins-debian1) Fri, 11 Feb 2022 16:23:08 +0000 swh-loader-svn (1.1.1-1~swh1) unstable-swh; urgency=medium * New upstream release 1.1.1 - (tagged by Antoine Lambert on 2022-01-28 11:42:12 +0100) * Upstream changes: - version 1.1.1 -- Software Heritage autobuilder (on jenkins-debian1) Fri, 28 Jan 2022 10:44:57 +0000 swh-loader-svn (1.1.0-1~swh1) unstable-swh; urgency=medium * New upstream release 1.1.0 - (tagged by Antoine Lambert on 2022-01-24 17:57:36 +0100) * Upstream changes: - version 1.1.0 -- Software Heritage autobuilder (on jenkins-debian1) Mon, 24 Jan 2022 17:01:15 +0000 swh-loader-svn (1.0.0-1~swh1) unstable-swh; urgency=medium * New upstream release 1.0.0 - (tagged by Antoine Lambert on 2022-01-19 16:51:43 +0100) * Upstream changes: - version 1.0.0 -- Software Heritage autobuilder (on jenkins-debian1) Wed, 19 Jan 2022 15:54:56 +0000 swh-loader-svn (0.10.2-1~swh1) unstable-swh; urgency=medium * New upstream release 0.10.2 - (tagged by Antoine Lambert on 2021-11-29 14:36:35 +0100) * Upstream changes: - version 0.10.2 -- Software Heritage autobuilder (on jenkins-debian1) Mon, 29 Nov 2021 13:40:41 +0000 swh-loader-svn (0.10.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.10.1 - (tagged by Antoine Lambert on 2021-11-25 15:20:27 +0100) * Upstream changes: - version 0.10.1 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 25 Nov 2021 14:26:34 +0000 swh-loader-svn (0.10.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.10.0 - (tagged by Antoine Lambert on 2021-11-19 11:58:43 +0100) * Upstream changes: - version 0.10.0 -- Software Heritage autobuilder (on jenkins-debian1) Fri, 19 Nov 2021 11:03:18 +0000 swh-loader-svn (0.9.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.9.1 - (tagged by Antoine R. Dumont (@ardumont) on 2021-11-18 14:19:23 +0100) * Upstream changes: - v0.9.1 - ra: Prevent erroneous removing of file state -- Software Heritage autobuilder (on jenkins-debian1) Thu, 18 Nov 2021 13:22:06 +0000 swh-loader-svn (0.9.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.9.0 - (tagged by Antoine Lambert on 2021-11-17 15:11:56 +0100) * Upstream changes: - version 0.9.0 -- Software Heritage autobuilder (on jenkins-debian1) Wed, 17 Nov 2021 14:14:34 +0000 swh-loader-svn (0.8.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.8.0 - (tagged by Antoine R. Dumont (@ardumont) on 2021-11-09 12:25:09 +0100) * Upstream changes: - v0.8.0 - SvnLoaderFromRemoteDump: Drop dump when svn repository is mounted - loader: Let log statement do the formatting - svn: Modify default loader to use the same as production task -- Software Heritage autobuilder (on jenkins-debian1) Tue, 09 Nov 2021 11:27:49 +0000 swh-loader-svn (0.7.3-1~swh1) unstable-swh; urgency=medium * New upstream release 0.7.3 - (tagged by Antoine R. Dumont (@ardumont) on 2021-11-08 17:29:05 +0100) * Upstream changes: - v0.7.3 - loader: Implement post_load to check for revision divergence - ra: Fix export of non link file with svn:special property set - loader: Fix error when first revision number is not one - loader: Perform a complete reloading when detecting altered history - ra: Handle invalid values for svn:eol-style property - ra: Handle unsetting svn:special property on svn links - ra: Do not process end of lines for svn links - ra: Fix edge case in svn:eol-style property handling - loader: Rename start_from_scratch parameter to incremental - loader: Let logging instructions do the formatting - loader: Fix SvnLoaderFromDumpArchive and add test - Remove reference to deprecated swh.model.identifiers module -- Software Heritage autobuilder (on jenkins-debian1) Mon, 08 Nov 2021 16:32:19 +0000 swh-loader-svn (0.7.2-1~swh1) unstable-swh; urgency=medium * New upstream release 0.7.2 - (tagged by Antoine R. Dumont (@ardumont) on 2021-09-30 10:07:44 +0200) * Upstream changes: - v0.7.2 - SvnLoaderFromRemoteDump: Fix failed visit which should be an uneventful visit - svn: Capture svnrdump failure as not_found instead of failed - mypy: Fix errors with release >= v0.900 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 30 Sep 2021 08:10:27 +0000 swh-loader-svn (0.7.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.7.1 - (tagged by Antoine Lambert on 2021-04-29 14:31:33 +0200) * Upstream changes: - version 0.7.1 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 29 Apr 2021 12:34:51 +0000 swh-loader-svn (0.7.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.7.0 - (tagged by Antoine R. Dumont (@ardumont) on 2021-02-17 18:19:35 +0100) * Upstream changes: - v0.7.0 - loader: Expect visit_date as an optional date in constructors - Rework loader instantiation logic according to loader core api -- Software Heritage autobuilder (on jenkins-debian1) Wed, 17 Feb 2021 17:23:16 +0000 swh-loader-svn (0.6.0-1~swh2) unstable-swh; urgency=medium * Update dependencies -- Antoine R. Dumont (@ardumont) Fri, 12 Feb 2021 18:05:59 +0100 swh-loader-svn (0.6.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.6.0 - (tagged by Antoine R. Dumont (@ardumont) on 2021-02-12 17:31:53 +0100) * Upstream changes: - v0.6.0 - svn.loader: Mark visit status as not_found when relevant - svn.loader: Mark visit status as failed - Drop the retrying dependency - tox.ini: pin black to the pre-commit version (19.10b0) to avoid flip-flops -- Software Heritage autobuilder (on jenkins-debian1) Fri, 12 Feb 2021 16:33:45 +0000 swh-loader-svn (0.5.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.5.0 - (tagged by Antoine R. Dumont (@ardumont) on 2020-10-02 14:16:50 +0200) * Upstream changes: - v0.5.0 - svn.loader: Migrate away from SWHConfig mixin - tests: Don't check the number of created 'person' objects - python: Reorder imports with isort - pre- commit: Add isort hook and configuration - pre-commit: Update flake8 hook configuration -- Software Heritage autobuilder (on jenkins-debian1) Fri, 02 Oct 2020 12:17:49 +0000 swh-loader-svn (0.4.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.4.1 - (tagged by Antoine Lambert on 2020-09-08 17:48:55 +0200) * Upstream changes: - version 0.4.1 -- Software Heritage autobuilder (on jenkins-debian1) Tue, 08 Sep 2020 15:53:11 +0000 swh-loader-svn (0.4.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.4.0 - (tagged by Antoine R. Dumont (@ardumont) on 2020-09-04 14:05:14 +0200) * Upstream changes: - v0.4.0 - Adapt storage.revision_get calls according to latest api change - Tell pytest not to recurse in dotdirs. -- Software Heritage autobuilder (on jenkins-debian1) Fri, 04 Sep 2020 12:07:00 +0000 swh-loader-svn (0.3.3-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.3 - (tagged by Antoine R. Dumont (@ardumont) on 2020-08-24 16:00:11 +0200) * Upstream changes: - v0.3.3 - svn.loader: Simplify and align incremental visit algorithm -- Software Heritage autobuilder (on jenkins-debian1) Mon, 24 Aug 2020 14:01:22 +0000 swh-loader-svn (0.3.2-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.2 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-28 08:07:12 +0200) * Upstream changes: - v0.3.2 - Migrate to setuptools-scm - Add missing conftest.py in MANIFEST.in -- Software Heritage autobuilder (on jenkins-debian1) Tue, 28 Jul 2020 06:09:53 +0000 swh-loader-svn (0.3.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.1 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-28 07:50:03 +0200) * Upstream changes: - v0.3.1 - loader: Update swh.storage.origin_get call to latest api change - tests: Reuse pytest fixtures from swh.loader.core - tests: Check against snapshot model object - svn.loader: Drop `build_swh_snapshot` function, oneliner used once - Fix branches types in tests - Clean up the swh.scheduler / swh.storage pytest plugin imports -- Software Heritage autobuilder (on jenkins-debian1) Tue, 28 Jul 2020 05:52:35 +0000 swh-loader-svn (0.3.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.0 - (tagged by David Douard on 2020-07-08 17:18:02 +0200) * Upstream changes: - v0.3.0 -- Software Heritage autobuilder (on jenkins-debian1) Wed, 08 Jul 2020 15:22:12 +0000 swh-loader-svn (0.2.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.2.1 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-03 15:17:44 +0200) * Upstream changes: - v0.2.1 - svn.loader: Fix uneventful visit case - svn.loader: Fix edge case on mistyped snapshot - tests: Refactor to use pytest - Reuse swh.model.from_disk.iter_directory function - Migrate to Use the object's object_type field when computing objects -- Software Heritage autobuilder (on jenkins-debian1) Fri, 03 Jul 2020 13:20:20 +0000 swh-loader-svn (0.2.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.2.0 - (tagged by Antoine R. Dumont (@ardumont) on 2020-06-23 15:25:21 +0200) * Upstream changes: - v0.2.0 - loader: Read snapshot out of the last origin visit status - tests: Refactor tests to use assert_last_visit_matches -- Software Heritage autobuilder (on jenkins-debian1) Tue, 23 Jun 2020 13:26:50 +0000 swh-loader-svn (0.1.2-1~swh1) unstable-swh; urgency=medium * New upstream release 0.1.2 - (tagged by Antoine R. Dumont (@ardumont) on 2020-06-09 11:03:56 +0200) * Upstream changes: - v0.1.2 - test_loader: assert returned loader status is what we expect - Adapt to swh.model 0.3 -- Software Heritage autobuilder (on jenkins-debian1) Tue, 09 Jun 2020 09:06:01 +0000 swh-loader-svn (0.1.1-1~swh3) unstable-swh; urgency=medium * Enable build for python3.8 (Debian ships a patched subvertpy.) -- Nicolas Dandrimont Wed, 03 Jun 2020 16:27:53 +0200 swh-loader-svn (0.1.1-1~swh2) unstable-swh; urgency=medium * Force removing __init__.py files. -- Nicolas Dandrimont Wed, 03 Jun 2020 16:13:31 +0200 swh-loader-svn (0.1.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.1.1 - (tagged by Antoine Lambert on 2020-06-03 11:49:31 +0200) * Upstream changes: - version 0.1.1 -- Software Heritage autobuilder (on jenkins-debian1) Wed, 03 Jun 2020 09:53:23 +0000 swh-loader-svn (0.1.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.1.0 - (tagged by Nicolas Dandrimont on 2020-06-02 18:54:30 +0200) * Upstream changes: - Release swh.loader.svn v0.1.0 - Blacken all files - Fix compatibility with swh.loader.core v0.1.0 - Fix parameter ordering for loader tasks -- Software Heritage autobuilder (on jenkins-debian1) Tue, 02 Jun 2020 16:58:55 +0000 swh-loader-svn (0.0.54-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.54 - (tagged by Antoine R. Dumont (@ardumont) on 2020-03-27 07:52:30 +0100) * Upstream changes: - v0.0.54 - svn.loader: Let core loader update snapshot & visit at the same time - test_loader: Add missing assertions on visit status - Fix crash on None snapshot -- Software Heritage autobuilder (on jenkins-debian1) Fri, 27 Mar 2020 06:57:15 +0000 swh-loader-svn (0.0.53-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.53 - (tagged by Antoine R. Dumont (@ardumont) on 2020-03-16 13:38:29 +0100) * Upstream changes: - v0.0.53 - Migrate to latest swh-loader- core api change - Move Person parsing to swh-model. -- Software Heritage autobuilder (on jenkins-debian1) Mon, 16 Mar 2020 12:41:14 +0000 swh-loader-svn (0.0.52-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.52 - (tagged by Antoine R. Dumont (@ardumont) on 2020-03-02 15:39:28 +0100) * Upstream changes: - v0.0.52 - loader.svn: Use swh.model objects within the loader - tox.ini: Add debugging dev tool in dedicated environment - converters: Clean up some more dead code - converters: Clean up dead code -- Software Heritage autobuilder (on jenkins-debian1) Mon, 02 Mar 2020 14:42:36 +0000 swh-loader-svn (0.0.51-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.51 - (tagged by Nicolas Dandrimont on 2020-02-19 20:01:31 +0100) * Upstream changes: - Release swh.loader.svn v0.0.51 - Ensure snapshots are properly linked by origin_visits - Update tests for swh.storage v0.0.174 -- Software Heritage autobuilder (on jenkins-debian1) Wed, 19 Feb 2020 19:20:03 +0000 swh-loader-svn (0.0.50-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.50 - (tagged by Antoine R. Dumont (@ardumont) on 2020-02-06 18:45:10 +0100) * Upstream changes: - v0.0.50 - loader.svn: Call storage's (skipped_)content_add endpoints - Fix sphinx warnings -- Software Heritage autobuilder (on jenkins-debian1) Thu, 06 Feb 2020 17:57:08 +0000 swh-loader-svn (0.0.49-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.49 - (tagged by Antoine R. Dumont (@ardumont) on 2020-01-15 15:59:52 +0100) * Upstream changes: - v0.0.49 - svn.loader: Adapt according to latest loader-core simplification -- Software Heritage autobuilder (on jenkins-debian1) Wed, 15 Jan 2020 15:03:50 +0000 swh-loader-svn (0.0.48-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.48 - (tagged by Antoine R. Dumont (@ardumont) on 2019-12-12 14:14:12 +0100) * Upstream changes: - v0.0.48 - svn.loader: Migrate to new buffer/filter behavior -- Software Heritage autobuilder (on jenkins-debian1) Thu, 12 Dec 2019 13:17:31 +0000 swh-loader-svn (0.0.47-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.47 - (tagged by Antoine R. Dumont (@ardumont) on 2019-12-12 11:18:45 +0100) * Upstream changes: - v0.0.47 - tasks: Enforce kwargs use in task message -- Software Heritage autobuilder (on jenkins-debian1) Thu, 12 Dec 2019 10:21:55 +0000 swh-loader-svn (0.0.46-1~swh2) unstable-swh; urgency=medium * Fix build for python3.7 only -- Antoine R. Dumont (@ardumont) Sat, 16 Feb 2019 02:43:37 +0100 swh-loader-svn (0.0.46-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.46 - (tagged by Antoine R. Dumont (@ardumont) on 2019-12-10 10:30:35 +0100) * Upstream changes: - v0.0.46 - tests: Migrate to use pytest- mock's mocker fixture - tasks: Use celery's shared_task decorator - loader.svn: Register svn worker - svn.tasks: Rename tasks according to production's - tasks: Adapt tasks according to the constructor unification - svn: Unify loader constructor with url as parameter - Fix a typo reported by codespell - Add a pre-commit config file - Migrate tox.ini to extras = xxx instead of deps = .[testing] - De-specify testenv:py3 - Include all requirements in MANIFEST.in -- Software Heritage autobuilder (on jenkins-debian1) Tue, 10 Dec 2019 09:38:45 +0000 swh-loader-svn (0.0.45-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.45 - (tagged by Stefano Zacchiroli on 2019-10-10 14:43:18 +0200) * Upstream changes: - v0.0.45 - * typing: minimal changes to make a no-op mypy run pass - * tox.ini: Fix py3 environment to use packaged tests - * Add attribute visit_type. - * Fix warning about pytest.mark.fs. - * add code of conduct document - * CONTRIBUTORS: add Ishan Bhanuka - * Remove unnecessary noqa - * Update dependency on swh-storage. - * Update argument name. - * Use origin urls instead of origin ids. - * update requirements - * Update coverage gitignore - * SvnLoaderFromRemoteDump: Add some debug output when svnrdump is running - * Remove debian packaging from master branch -- Software Heritage autobuilder (on jenkins-debian1) Thu, 10 Oct 2019 12:46:37 +0000 swh-loader-svn (0.0.44-1~swh3) unstable-swh; urgency=low * New upstream release (fixing the distribution) -- Antoine Romain Dumont Sat, 16 Feb 2019 02:43:37 +0100 swh-loader-svn (0.0.44-1~swh2) unstable; urgency=low * New upstream release to fix debian packaging -- Antoine Romain Dumont Sat, 16 Feb 2019 02:19:27 +0100 swh-loader-svn (0.0.44-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.44 - (tagged by David Douard on 2019-02-07 17:35:43 +0100) * Upstream changes: - v0.0.44 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 07 Feb 2019 16:41:45 +0000 swh-loader-svn (0.0.42-1~swh1) unstable-swh; urgency=medium * v0.0.43 * loader: Rename default branch as HEAD * svn.tests: Refactor using loader-core mixins * setup: prepare for pypi upload * Add requirements-test.txt * docs: add title and brief module description -- Antoine R. Dumont (@ardumont) Thu, 11 Oct 2018 14:52:54 +0200 swh-loader-svn (0.0.41-1~swh1) unstable-swh; urgency=medium * v0.0.41 * Fix UnicodeDecodeError when decoding user-defined svn properties * preventing the origin ingestion to finish (T946) * Fix CR/LF issue preventing the origin ingestion to finish (T570) * svn.loader: Add more efficient loader based on remote dumps * svn.loader: Refactoring (align loader with other, name conventions, * improve maintenance, simplify some code, etc...) -- Antoine R. Dumont (@ardumont) Fri, 28 Sep 2018 12:16:31 +0200 swh-loader-svn (0.0.40-1~swh1) unstable-swh; urgency=medium * v0.0.40 * svn/loader: Remove wrong parameter name in dangling clean up call * svn/loader: Refactor: Reuse loader.core.utils.clean_dangling_folders * svn/tasks: Explicit tasks' arguments * svn/loader: Add pre-cleanup step to clean potential dangling folders * d/control: Bump to latest python3-swh.loader.core -- Antoine R. Dumont (@ardumont) Fri, 09 Mar 2018 14:51:09 +0100 swh-loader-svn (0.0.39-1~swh1) unstable-swh; urgency=medium * v0.0.39 * Improve origin_visit initialization step * Properly sandbox the prepare statement so that if it breaks, we can * update appropriately the visit with the correct status -- Antoine R. Dumont (@ardumont) Wed, 07 Mar 2018 11:31:34 +0100 swh-loader-svn (0.0.38-1~swh1) unstable-swh; urgency=medium * v0.0.38 * In failure mode, try harder to make snapshot -- Antoine R. Dumont (@ardumont) Fri, 09 Feb 2018 14:13:30 +0100 swh-loader-svn (0.0.37-1~swh1) unstable-swh; urgency=medium * v0.0.37 * swh.loader.svn: Detect eol inconsistency and make the loading fail * swh.loader.svn: Reduce log verbosity * swh.loader.svn: Migrate exception to their specific module -- Antoine R. Dumont (@ardumont) Thu, 08 Feb 2018 17:57:15 +0100 swh-loader-svn (0.0.36-1~swh1) unstable-swh; urgency=medium * v0.0.36 * swh.loader.svn: Fix corner edge case on symbolic link * swh.loader.svn: Deal with empty svn repository edge case -- Antoine R. Dumont (@ardumont) Wed, 07 Feb 2018 12:36:45 +0100 swh-loader-svn (0.0.35-1~swh1) unstable-swh; urgency=medium * Release swh.loader.svn v0.0.35 * Update to use snapshots instead of occurrences * Properly return loader status to scheduler -- Nicolas Dandrimont Tue, 06 Feb 2018 14:40:28 +0100 swh-loader-svn (0.0.34-1~swh1) unstable-swh; urgency=medium * v0.0.34 * Unify temporary directory naming -- Antoine R. Dumont (@ardumont) Fri, 15 Dec 2017 12:26:50 +0100 swh-loader-svn (0.0.33-1~swh1) unstable-swh; urgency=medium * v0.0.33 * swh.loader.svn: Add option to load a repository from scratch * swh.loader.svn.loader: Fix special symlink case -- Antoine R. Dumont (@ardumont) Fri, 08 Dec 2017 17:38:37 +0100 swh-loader-svn (0.0.32-1~swh1) unstable-swh; urgency=medium * v0.0.32 * d/control: Bump dependency to latest swh-loader-core * tests: Add commits with wrong symbolic links (broken, empty space) * ra: Fix not cleaned up path leading to file exists error * docs: add absolute anchor to documentation index -- Antoine R. Dumont (@ardumont) Wed, 08 Nov 2017 13:22:42 +0100 swh-loader-svn (0.0.31-1~swh1) unstable-swh; urgency=medium * v0.0.31 * swh.loader.svn.ra: Fix blank space in symlink path split problem * swh.loader.svn.ra: Fix missing case about existing directory * swh.loader.svn.ra: Explicit the closure * swh.loader.svn: Deal with origin creation inhibition in base class * swh.loader.svn: Remove unneeded instruction * swh.loader.svn: Fix initializing the origin multiple times * swh.loader.svn: Remove unneeded except clause * swh.loader.svn: uneventful visit is not an error -- Antoine R. Dumont (@ardumont) Tue, 24 Oct 2017 16:35:01 +0200 swh-loader-svn (0.0.30-1~swh1) unstable-swh; urgency=medium * Release swh.loader.svn version 0.0.30 * Update packaging runes -- Nicolas Dandrimont Thu, 12 Oct 2017 18:07:58 +0200 swh-loader-svn (0.0.29-1~swh1) unstable-swh; urgency=medium * v0.0.29 * swh.loader.svn.loader: Fix missing revision id key (which can lead * to an empty partial visit for example in the case of a repository * holding an svn:externals property). -- Antoine R. Dumont (@ardumont) Mon, 09 Oct 2017 16:02:23 +0200 swh-loader-svn (0.0.28-1~swh1) unstable-swh; urgency=medium * Release swh.loader.svn v0.0.28 * Replace swh.model.git with swh.model.from_disk -- Nicolas Dandrimont Fri, 06 Oct 2017 17:18:57 +0200 swh-loader-svn (0.0.27-1~swh1) unstable-swh; urgency=medium * v0.0.27 * d/control: Bump dependency to latest swh.loader.core * swh.loader.svn: Add args to raised exception in edge cases * swh-hashtree: Fix no longer existing swh.core.hashutil import * docs/: add sphinx apidoc generation skeleton -- Antoine R. Dumont (@ardumont) Tue, 03 Oct 2017 19:01:26 +0200 swh-loader-svn (0.0.26-1~swh1) unstable-swh; urgency=medium * Release swh.loader.svn 0.0.26 * update tasks to new swh.scheduler API -- Nicolas Dandrimont Mon, 12 Jun 2017 18:25:40 +0200 swh-loader-svn (0.0.25-1~swh1) unstable-swh; urgency=medium * v0.0.25 * Add blake2s256 new hash computation on content * Use gzip instead of pigz * Migrate from swh.core.hashutil to swh.model.hashutil -- Antoine R. Dumont (@ardumont) Tue, 04 Apr 2017 14:18:57 +0200 swh-loader-svn (0.0.24-1~swh1) unstable-swh; urgency=medium * v0.0.24 * Simplify loader's logic -- Antoine R. Dumont (@ardumont) Wed, 22 Feb 2017 14:09:15 +0100 swh-loader-svn (0.0.23-1~swh1) unstable-swh; urgency=medium * Release swh.loader.svn v0.0.23 * Return timestamps as pairs of integers (Close T680) -- Nicolas Dandrimont Tue, 14 Feb 2017 19:40:27 +0100 swh-loader-svn (0.0.22-1~swh1) unstable-swh; urgency=medium * v0.0.22 * Pass optionally the visit date to producer -- Antoine R. Dumont (@ardumont) Tue, 10 Jan 2017 15:03:45 +0100 swh-loader-svn (0.0.21-1~swh1) unstable-swh; urgency=medium * v0.0.21 * Use the dump's fetch date as visit date -- Antoine R. Dumont (@ardumont) Tue, 10 Jan 2017 15:03:24 +0100 swh-loader-svn (0.0.20-1~swh1) unstable-swh; urgency=medium * v0.0.20 * Fix function call -- Antoine R. Dumont (@ardumont) Fri, 06 Jan 2017 15:38:14 +0100 swh-loader-svn (0.0.19-1~swh1) unstable-swh; urgency=medium * v0.0.19 * Rename task_queue adequately * Fix - Remove empty temporary directory in corner cases * Fix generator issue when using local storage -- Antoine R. Dumont (@ardumont) Fri, 06 Jan 2017 14:10:19 +0100 swh-loader-svn (0.0.18-1~swh1) unstable-swh; urgency=medium * v0.0.18 * Add check revision hash computation divergence during loading * Add task to load a svn repository from an archive dump and load the * repository * Raise error when svn:externals keyword is encountered (T610) since * it's not dealt with yet. * Revert modifications related to svn:eol-style since no good policy has * been found yet to deal with it -- Antoine R. Dumont (@ardumont) Thu, 15 Dec 2016 10:57:26 +0100 swh-loader-svn (0.0.17-1~swh1) unstable-swh; urgency=medium * v0.0.17 * Force eol conversion when needed -- Antoine R. Dumont (@ardumont) Thu, 08 Dec 2016 18:25:55 +0100 swh-loader-svn (0.0.16-1~swh1) unstable-swh; urgency=medium * v0.0.16 * Improve origin_visit policy -- Antoine R. Dumont (@ardumont) Fri, 26 Aug 2016 15:59:31 +0200 swh-loader-svn (0.0.15-1~swh1) unstable-swh; urgency=medium * v0.0.15 * Update svn loader to register origin_visit's state -- Antoine R. Dumont (@ardumont) Wed, 24 Aug 2016 14:56:54 +0200 swh-loader-svn (0.0.14-1~swh1) unstable-swh; urgency=medium * v0.0.14 * Update svn loader to register origin_visit's state -- Antoine R. Dumont (@ardumont) Tue, 23 Aug 2016 16:38:16 +0200 swh-loader-svn (0.0.13-1~swh1) unstable-swh; urgency=medium * v0.0.13 -- Antoine R. Dumont (@ardumont) Thu, 18 Aug 2016 10:30:13 +0200 swh-loader-svn (0.0.12-1~swh1) unstable-swh; urgency=medium * v0.0.12 * Inhibit keyword expansion during export phase (svn update) -- Antoine R. Dumont (@ardumont) Wed, 06 Jul 2016 16:17:51 +0200 swh-loader-svn (0.0.11-1~swh1) unstable-swh; urgency=medium * v0.0.11 * Reschedule when error on loading an svn repository * Start from last known revision * Permit to load local mirror with remote mirror origin -- Antoine R. Dumont (@ardumont) Fri, 01 Jul 2016 16:51:37 +0200 swh-loader-svn (0.0.10-1~swh1) unstable-swh; urgency=medium * v0.0.10 * Add retry policy around britle action (connection to remote svn server) -- Antoine R. Dumont (@ardumont) Sat, 25 Jun 2016 11:30:34 +0200 swh-loader-svn (0.0.9-1~swh1) unstable-swh; urgency=medium * v0.0.9 * Restrict loader to 2 possible policies: swh, git-svn * Improve coverage * Update README -- Antoine R. Dumont (@ardumont) Fri, 24 Jun 2016 13:56:54 +0200 swh-loader-svn (0.0.8-1~swh1) unstable-swh; urgency=medium * v0.0.8 * Use remote-access approach to hash and compute hashes from svn * repository (migrate major dependency from pysvn to subvertpy) -- Antoine R. Dumont (@ardumont) Sat, 11 Jun 2016 19:55:54 +0200 swh-loader-svn (0.0.7-1~swh1) unstable-swh; urgency=medium * v0.0.7 * d/control: Bump dependency version to latest swh-model -- Antoine R. Dumont (@ardumont) Thu, 26 May 2016 16:10:32 +0200 swh-loader-svn (0.0.6-1~swh1) unstable-swh; urgency=medium * v0.0.6 * d/control: Bump swh-model dependency * Simplify ignore folder corner cases * Rework log -- Antoine R. Dumont (@ardumont) Wed, 25 May 2016 23:53:10 +0200 swh-loader-svn (0.0.5-1~swh1) unstable-swh; urgency=medium * v0.0.5 * Add svn update behavior as option (default to True) * d/control: Bump dependency version to swh-loader-core -- Antoine R. Dumont (@ardumont) Wed, 25 May 2016 12:44:35 +0200 swh-loader-svn (0.0.4-1~swh1) unstable-swh; urgency=medium * v0.0.4 * Add options for git-svn like behavior (git svn clone --no- metadata) * Ignore empty folder * Add extra-line in svn commit message * No revision metadata (beware that this break the update repo * mechanism) * Also, use @ pattern for committer/author -- Antoine R. Dumont (@ardumont) Tue, 24 May 2016 17:04:40 +0200 swh-loader-svn (0.0.3-1~swh1) unstable-swh; urgency=medium * v0.0.3 * Improve filesystem cleanup (before: left empty directory) * Use pysvn to extract uuid information (before: subprocess with subversion call) * Keep the working copy's name on the local checkout -- Antoine R. Dumont (@ardumont) Fri, 15 Apr 2016 16:10:51 +0200 swh-loader-svn (0.0.2-1~swh1) unstable-swh; urgency=medium * v0.0.2 * Update to the right swh dependencies. -- Antoine R. Dumont (@ardumont) Fri, 15 Apr 2016 13:02:04 +0200 swh-loader-svn (0.0.1-1~swh1) unstable-swh; urgency=medium * Initial release * v0.0.1 -- Antoine R. Dumont (@ardumont) Wed, 13 Apr 2016 16:46:37 +0200 diff --git a/swh.loader.svn.egg-info/PKG-INFO b/swh.loader.svn.egg-info/PKG-INFO index 233b713..164e16d 100644 --- a/swh.loader.svn.egg-info/PKG-INFO +++ b/swh.loader.svn.egg-info/PKG-INFO @@ -1,61 +1,61 @@ Metadata-Version: 2.1 Name: swh.loader.svn -Version: 1.3.1 +Version: 1.3.2 Summary: Software Heritage Loader SVN Home-page: https://forge.softwareheritage.org/diffusion/DLDSVN Author: Software Heritage developers Author-email: swh-devel@inria.fr License: UNKNOWN Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest Project-URL: Funding, https://www.softwareheritage.org/donate Project-URL: Source, https://forge.softwareheritage.org/source/swh-loader-svn Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-loader-svn/ Platform: UNKNOWN Classifier: Programming Language :: Python :: 3 Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Classifier: Operating System :: OS Independent Classifier: Development Status :: 5 - Production/Stable Requires-Python: >=3.7 Description-Content-Type: text/markdown Provides-Extra: testing License-File: LICENSE License-File: AUTHORS swh-loader-svn ============== The Software Heritage SVN Loader is a tool and a library to walk a remote svn repository and inject into the SWH dataset all contained files that weren't known before. The main entry points are - :class:`swh.loader.svn.loader.SvnLoader` for the main svn loader which ingests content out of a remote svn repository - :class:`swh.loader.svn.loader.SvnLoaderFromDumpArchive` which mounts a repository out of a svn dump prior to ingest it. - :class:`swh.loader.svn.loader.SvnLoaderFromRemoteDump` which mounts a repository with svnrdump prior to ingest its content. # CLI run With the configuration: /tmp/loader_svn.yml: ``` storage: cls: remote args: url: http://localhost:5002/ ``` Run: ``` swh loader --config-file /tmp/loader_svn.yml \ run svn ``` diff --git a/swh/loader/svn/loader.py b/swh/loader/svn/loader.py index 7e1839e..2d898ed 100644 --- a/swh/loader/svn/loader.py +++ b/swh/loader/svn/loader.py @@ -1,821 +1,820 @@ # Copyright (C) 2015-2021 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU General Public License version 3, or any later version # See top-level LICENSE file for more information """Loader in charge of injecting either new or existing svn mirrors to swh-storage. """ from datetime import datetime from mmap import ACCESS_WRITE, mmap import os import pty import re import shutil from subprocess import Popen import tempfile from typing import Any, Dict, Iterator, List, Optional, Sequence, Tuple from subvertpy import SubversionException from swh.loader.core.loader import BaseLoader from swh.loader.core.utils import clean_dangling_folders from swh.loader.exception import NotFound from swh.loader.svn.svn import SvnRepo from swh.model import from_disk, hashutil from swh.model.model import ( Content, Directory, Revision, SkippedContent, Snapshot, SnapshotBranch, TargetType, ) from swh.storage.algos.snapshot import snapshot_get_latest from swh.storage.interface import StorageInterface from . import converters from .exception import SvnLoaderHistoryAltered, SvnLoaderUneventful from .utils import ( OutputStream, init_svn_repo_from_archive_dump, init_svn_repo_from_dump, ) DEFAULT_BRANCH = b"HEAD" TEMPORARY_DIR_PREFIX_PATTERN = "swh.loader.svn." SUBVERSION_ERROR = re.compile(r".*(E[0-9]{6}):.*") SUBVERSION_NOT_FOUND = "E170013" class SvnLoader(BaseLoader): """SVN loader. The repository is either remote or local. The loader deals with update on an already previously loaded repository. """ visit_type = "svn" def __init__( self, storage: StorageInterface, url: str, origin_url: Optional[str] = None, visit_date: Optional[datetime] = None, incremental: bool = True, temp_directory: str = "/tmp", debug: bool = False, check_revision: int = 0, **kwargs: Any, ): """Load a svn repository (either remote or local). Args: url: The default origin url origin_url: Optional original url override to use as origin reference in the archive. If not provided, "url" is used as origin. visit_date: Optional date to override the visit date incremental: If True, the default, starts from the last snapshot (if any). Otherwise, starts from the initial commit of the repository. temp_directory: The temporary directory to use as root directory for working directory computations debug: If true, run the loader in debug mode. At the end of the loading, the temporary working directory is not cleaned up to ease inspection. Defaults to false. check_revision: The number of svn commits between checks for hash divergence - max_content_size: Default max content size allowed """ # technical svn uri to act on svn repository self.svn_url = url # origin url as unique identifier for origin in swh archive origin_url = origin_url or self.svn_url super().__init__(storage=storage, origin_url=origin_url, **kwargs) self.debug = debug self.temp_directory = temp_directory self.done = False self.svnrepo = None self.skip_post_load = False # Revision check is configurable self.check_revision = check_revision # internal state used to store swh objects self._contents: List[Content] = [] self._skipped_contents: List[SkippedContent] = [] self._directories: List[Directory] = [] self._revisions: List[Revision] = [] self._snapshot: Optional[Snapshot] = None # internal state, current visit self._last_revision = None self._visit_status = "full" self._load_status = "uneventful" self.visit_date = visit_date or self.visit_date self.incremental = incremental self.snapshot: Optional[Snapshot] = None # state from previous visit self.latest_snapshot = None self.latest_revision: Optional[Revision] = None self.from_dump = False def pre_cleanup(self): """Cleanup potential dangling files from prior runs (e.g. OOM killed tasks) """ clean_dangling_folders( self.temp_directory, pattern_check=TEMPORARY_DIR_PREFIX_PATTERN, log=self.log, ) def cleanup(self): """Clean up the svn repository's working representation on disk.""" if not self.svnrepo: # could happen if `prepare` fails return if self.debug: self.log.error( """NOT FOR PRODUCTION - debug flag activated Local repository not cleaned up for investigation: %s""", self.svnrepo.local_url.decode("utf-8"), ) return self.svnrepo.clean_fs() def swh_revision_hash_tree_at_svn_revision(self, revision: int) -> bytes: """Compute and return the hash tree at a given svn revision. Args: rev: the svn revision we want to check Returns: The hash tree directory as bytes. """ assert self.svnrepo is not None local_dirname, local_url = self.svnrepo.export_temporary(revision) root_dir = from_disk.Directory.from_disk(path=local_url) self.svnrepo.clean_fs(local_dirname) return root_dir.hash def _latest_snapshot_revision( self, origin_url: str, ) -> Optional[Tuple[Snapshot, Revision]]: """Look for latest snapshot revision and returns it if any. Args: origin_url: Origin identifier previous_swh_revision: possible previous swh revision (either a dict or revision identifier) Returns: Tuple of the latest Snapshot from the previous visit and its targeted revision if any or None otherwise. """ storage = self.storage latest_snapshot = snapshot_get_latest(storage, origin_url) if not latest_snapshot: return None branches = latest_snapshot.branches if not branches: return None branch = branches.get(DEFAULT_BRANCH) if not branch: return None if branch.target_type != TargetType.REVISION: return None swh_id = branch.target revision = storage.revision_get([swh_id])[0] if not revision: return None return latest_snapshot, revision def build_swh_revision( self, rev: int, commit: Dict, dir_id: bytes, parents: Sequence[bytes] ) -> Revision: """Build the swh revision dictionary. This adds: - the `'synthetic`' flag to true - the '`extra_headers`' containing the repository's uuid and the svn revision number. Args: rev: the svn revision number commit: the commit data: revision id, date, author, and message dir_id: the upper tree's hash identifier parents: the parents' identifiers Returns: The swh revision corresponding to the svn revision. """ assert self.svnrepo is not None return converters.build_swh_revision( rev, commit, self.svnrepo.uuid, dir_id, parents ) def check_history_not_altered(self, revision_start: int, swh_rev: Revision) -> bool: """Given a svn repository, check if the history was modified in between visits.""" revision_id = swh_rev.id parents = swh_rev.parents assert self.svnrepo is not None commit, root_dir = self.svnrepo.swh_hash_data_at_revision(revision_start) dir_id = root_dir.hash swh_revision = self.build_swh_revision(revision_start, commit, dir_id, parents) swh_revision_id = swh_revision.id return swh_revision_id == revision_id def start_from(self) -> Tuple[int, int]: """Determine from where to start the loading. Returns: tuple (revision_start, revision_end) Raises: SvnLoaderHistoryAltered: When a hash divergence has been detected (should not happen) SvnLoaderUneventful: Nothing changed since last visit """ assert self.svnrepo is not None, "svnrepo initialized in the `prepare` method" revision_head = self.svnrepo.head_revision() if revision_head == 0: # empty repository case revision_start = 0 revision_end = 0 else: # default configuration revision_start = self.svnrepo.initial_revision() revision_end = revision_head # start from a previous revision if any if self.incremental and self.latest_revision is not None: extra_headers = dict(self.latest_revision.extra_headers) revision_start = int(extra_headers[b"svn_revision"]) if not self.check_history_not_altered(revision_start, self.latest_revision): self.log.debug( ( "History of svn %s@%s altered. " "A complete reloading of the repository will be performed." ), self.svnrepo.remote_url, revision_start, ) revision_start = 0 # now we know history is ok, we start at next revision revision_start = revision_start + 1 if revision_start > revision_end: msg = "%s@%s already injected." % (self.svnrepo.remote_url, revision_end) raise SvnLoaderUneventful(msg) self.log.info( "Processing revisions [%s-%s] for %s", revision_start, revision_end, self.svnrepo, ) return revision_start, revision_end def _check_revision_divergence(self, rev: int, dir_id: bytes) -> None: """Check for hash revision computation divergence. The Rationale behind this is that svn can trigger unknown edge cases (mixed CRLF, svn properties, etc...). Those are not always easy to spot. Adding a regular check will help spotting potential missing edge cases. Args: rev: The actual revision we are computing from dir_id: The actual directory for the given revision Raises ValueError if a hash divergence is detected """ self.log.debug("Checking hash computations on revision %s...", rev) checked_dir_id = self.swh_revision_hash_tree_at_svn_revision(rev) if checked_dir_id != dir_id: err = ( "Hash tree computation divergence detected " "(%s != %s), stopping!" % ( hashutil.hash_to_hex(dir_id), hashutil.hash_to_hex(checked_dir_id), ) ) raise ValueError(err) def process_svn_revisions( self, svnrepo, revision_start, revision_end ) -> Iterator[ Tuple[List[Content], List[SkippedContent], List[Directory], Revision] ]: """Process svn revisions from revision_start to revision_end. At each svn revision, apply new diffs and simultaneously compute swh hashes. This yields those computed swh hashes as a tuple (contents, directories, revision). Note that at every `self.check_revision`, a supplementary check takes place to check for hash-tree divergence (related T570). Yields: tuple (contents, directories, revision) of dict as a dictionary with keys, sha1_git, sha1, etc... Raises: ValueError in case of a hash divergence detection """ gen_revs = svnrepo.swh_hash_data_per_revision(revision_start, revision_end) parents = (self.latest_revision.id,) if self.latest_revision is not None else () count = 0 for rev, commit, new_objects, root_directory in gen_revs: count += 1 # Send the associated contents/directories _contents, _skipped_contents, _directories = new_objects # compute the fs tree's checksums dir_id = root_directory.hash swh_revision = self.build_swh_revision(rev, commit, dir_id, parents) self.log.debug( "rev: %s, swhrev: %s, dir: %s", rev, hashutil.hash_to_hex(swh_revision.id), hashutil.hash_to_hex(dir_id), ) if ( self.check_revision and self.check_revision != 0 and count % self.check_revision == 0 ): self._check_revision_divergence(rev, dir_id) parents = (swh_revision.id,) yield _contents, _skipped_contents, _directories, swh_revision if not self.debug and self.svnrepo: # clean directory where revisions were replayed to gain some disk space # before the post_load operation self.svnrepo.clean_fs(self.svnrepo.local_url) def svn_repo(self, *args, **kwargs): """Wraps the creation of SvnRepo object and handles not found repository errors.""" try: return SvnRepo(*args, **kwargs) except SubversionException as e: error_msgs = [ "Unable to connect to a repository at URL", "Unknown URL type", ] for msg in error_msgs: if msg in e.args[0]: self._load_status = "uneventful" raise NotFound(e) raise def prepare(self): if self.incremental: latest_snapshot_revision = self._latest_snapshot_revision(self.origin.url) if latest_snapshot_revision: self.latest_snapshot, self.latest_revision = latest_snapshot_revision self._snapshot = self.latest_snapshot self._last_revision = self.latest_revision local_dirname = self._create_tmp_dir(self.temp_directory) self.svnrepo = self.svn_repo( self.svn_url, self.origin.url, local_dirname, self.max_content_size, self.from_dump, ) try: revision_start, revision_end = self.start_from() self.swh_revision_gen = self.process_svn_revisions( self.svnrepo, revision_start, revision_end ) except SvnLoaderUneventful as e: self.log.warning(e) self.done = True self._load_status = "uneventful" except SvnLoaderHistoryAltered as e: self.log.error(e) self.done = True self._visit_status = "partial" def fetch_data(self): """Fetching svn revision information. This will apply svn revision as patch on disk, and at the same time, compute the swh hashes. In effect, fetch_data fetches those data and compute the necessary swh objects. It's then stored in the internal state instance variables (initialized in `_prepare_state`). This is up to `store_data` to actually discuss with the storage to store those objects. Returns: bool: True to continue fetching data (next svn revision), False to stop. """ if self.done: return False try: data = next(self.swh_revision_gen) self._load_status = "eventful" except StopIteration: self.done = True # Stopping iteration self._visit_status = "full" except Exception as e: # svn:external, hash divergence, i/o error... self.log.exception(e) self.done = True # Stopping iteration self._visit_status = "partial" else: self._contents, self._skipped_contents, self._directories, rev = data if rev: self._last_revision = rev self._revisions.append(rev) return not self.done def store_data(self): """We store the data accumulated in internal instance variable. If the iteration over the svn revisions is done, we create the snapshot and flush to storage the data. This also resets the internal instance variable state. """ self.storage.skipped_content_add(self._skipped_contents) self.storage.content_add(self._contents) self.storage.directory_add(self._directories) self.storage.revision_add(self._revisions) if self.done: # finish line, snapshot! self.snapshot = self.generate_and_load_snapshot( revision=self._last_revision, snapshot=self._snapshot ) self.flush() self.loaded_snapshot_id = self.snapshot.id # reset internal state for next iteration self._revisions = [] def generate_and_load_snapshot( self, revision: Optional[Revision] = None, snapshot: Optional[Snapshot] = None ) -> Snapshot: """Create the snapshot either from existing revision or snapshot. Revision (supposedly new) has priority over the snapshot (supposedly existing one). Args: revision (dict): Last revision seen if any (None by default) snapshot (dict): Snapshot to use if any (None by default) Returns: Optional[Snapshot] The newly created snapshot """ if revision: # Priority to the revision snap = Snapshot( branches={ DEFAULT_BRANCH: SnapshotBranch( target=revision.id, target_type=TargetType.REVISION ) } ) elif snapshot: # Fallback to prior snapshot snap = snapshot else: raise ValueError( "generate_and_load_snapshot called with null revision and snapshot!" ) self.log.debug("snapshot: %s", snap) self.storage.snapshot_add([snap]) return snap def load_status(self): return { "status": self._load_status, } def visit_status(self): return self._visit_status def post_load(self, success: bool = True) -> None: if self.skip_post_load: return if success and self._last_revision is not None: # check if the reconstructed filesystem for the last loaded revision is # consistent with the one obtained with a svn export operation. If it is not # the case, an exception will be raised to report the issue and mark the # visit as partial self._check_revision_divergence( int(dict(self._last_revision.extra_headers)[b"svn_revision"]), self._last_revision.directory, ) def _create_tmp_dir(self, root_tmp_dir: str) -> str: return tempfile.mkdtemp( dir=root_tmp_dir, prefix=TEMPORARY_DIR_PREFIX_PATTERN, suffix="-%s" % os.getpid(), ) class SvnLoaderFromDumpArchive(SvnLoader): """Uncompress an archive containing an svn dump, mount the svn dump as a local svn repository and load that repository. """ def __init__( self, storage: StorageInterface, url: str, archive_path: str, origin_url: Optional[str] = None, incremental: bool = False, visit_date: Optional[datetime] = None, temp_directory: str = "/tmp", debug: bool = False, check_revision: int = 0, - max_content_size: Optional[int] = None, + **kwargs: Any, ): super().__init__( storage=storage, url=url, origin_url=origin_url, incremental=incremental, visit_date=visit_date, temp_directory=temp_directory, debug=debug, check_revision=check_revision, - max_content_size=max_content_size, + **kwargs, ) self.archive_path = archive_path self.temp_dir = None self.repo_path = None self.from_dump = True def prepare(self): self.log.info("Archive to mount and load %s", self.archive_path) self.temp_dir, self.repo_path = init_svn_repo_from_archive_dump( self.archive_path, prefix=TEMPORARY_DIR_PREFIX_PATTERN, suffix="-%s" % os.getpid(), root_dir=self.temp_directory, ) self.svn_url = f"file://{self.repo_path}" super().prepare() def cleanup(self): super().cleanup() if self.temp_dir and os.path.exists(self.temp_dir): self.log.debug( "Clean up temporary directory dump %s for project %s", self.temp_dir, os.path.basename(self.repo_path), ) shutil.rmtree(self.temp_dir) class SvnLoaderFromRemoteDump(SvnLoader): """Create a subversion repository dump out of a remote svn repository (using the svnrdump utility). Then, mount the repository locally and load that repository. """ def __init__( self, storage: StorageInterface, url: str, origin_url: Optional[str] = None, incremental: bool = True, visit_date: Optional[datetime] = None, temp_directory: str = "/tmp", debug: bool = False, check_revision: int = 0, - max_content_size: Optional[int] = None, + **kwargs: Any, ): super().__init__( storage=storage, url=url, origin_url=origin_url, incremental=incremental, visit_date=visit_date, temp_directory=temp_directory, debug=debug, check_revision=check_revision, - max_content_size=max_content_size, + **kwargs, ) self.from_dump = True self.temp_dir = self._create_tmp_dir(self.temp_directory) self.repo_path = None self.truncated_dump = False def get_last_loaded_svn_rev(self, svn_url: str) -> int: """Check if the svn repository has already been visited and return the last loaded svn revision number or -1 otherwise. """ origin = list(self.storage.origin_get([svn_url]))[0] if not origin: return -1 svn_revision = -1 try: latest_snapshot_revision = self._latest_snapshot_revision(origin.url) if latest_snapshot_revision: _, latest_revision = latest_snapshot_revision latest_revision_headers = dict(latest_revision.extra_headers) svn_revision = int(latest_revision_headers[b"svn_revision"]) except Exception: pass return svn_revision def dump_svn_revisions(self, svn_url: str, last_loaded_svn_rev: int = -1) -> str: """Generate a subversion dump file using the svnrdump tool. If the svnrdump command failed somehow, the produced dump file is analyzed to determine if a partial loading is still feasible. Raises: NotFound when the repository is no longer found at url Returns: The dump_path of the repository mounted """ # Build the svnrdump command line svnrdump_cmd = ["svnrdump", "dump", svn_url] # Launch the svnrdump command while capturing stderr as # successfully dumped revision numbers are printed to it dump_temp_dir = tempfile.mkdtemp(dir=self.temp_dir) dump_name = "".join(c for c in svn_url if c.isalnum()) dump_path = "%s/%s.svndump" % (dump_temp_dir, dump_name) stderr_lines = [] self.log.debug("Executing %s", " ".join(svnrdump_cmd)) with open(dump_path, "wb") as dump_file: stderr_r, stderr_w = pty.openpty() svnrdump = Popen(svnrdump_cmd, stdout=dump_file, stderr=stderr_w) os.close(stderr_w) stderr_stream = OutputStream(stderr_r) readable = True error_codes: List[str] = [] error_messages: List[str] = [] while readable: lines, readable = stderr_stream.read_lines() stderr_lines += lines for line in lines: self.log.debug(line) match = SUBVERSION_ERROR.search(line) if match: error_codes.append(match.group(1)) error_messages.append(line) svnrdump.wait() os.close(stderr_r) if svnrdump.returncode == 0: return dump_path # There was an error but it does not mean that no revisions # can be loaded. # Get the stderr line with latest dumped revision last_dumped_rev = None for stderr_line in reversed(stderr_lines): if stderr_line.startswith("* Dumped revision"): last_dumped_rev = stderr_line break if last_dumped_rev: # Get the latest dumped revision number matched_rev = re.search(".*revision ([0-9]+)", last_dumped_rev) last_dumped_rev = int(matched_rev.group(1)) if matched_rev else -1 # Check if revisions inside the dump file can be loaded anyway if last_dumped_rev > last_loaded_svn_rev: self.log.debug( ( "svnrdump did not dump all expected revisions " "but revisions range %s:%s are available in " "the generated dump file and will be loaded " "into the archive." ), last_loaded_svn_rev + 1, last_dumped_rev, ) # Truncate the dump file after the last successfully dumped # revision to avoid the loading of corrupted data self.log.debug( ( "Truncating dump file after the last " "successfully dumped revision (%s) to avoid " "the loading of corrupted data" ), last_dumped_rev, ) with open(dump_path, "r+b") as f: with mmap(f.fileno(), 0, access=ACCESS_WRITE) as s: pattern = ( "Revision-number: %s" % (last_dumped_rev + 1) ).encode() n = s.rfind(pattern) if n != -1: s.resize(n) self.truncated_dump = True return dump_path elif last_dumped_rev != -1 and last_dumped_rev < last_loaded_svn_rev: raise Exception( ( "Last dumped subversion revision (%s) is " "lesser than the last one loaded into the " "archive (%s)." ) % (last_dumped_rev, last_loaded_svn_rev) ) if SUBVERSION_NOT_FOUND in error_codes: raise NotFound( f"{SUBVERSION_NOT_FOUND}: Repository never existed or disappeared" ) raise Exception( "An error occurred when running svnrdump and " "no exploitable dump file has been generated.\n" + "\n".join(error_messages) ) def prepare(self): # First, check if previous revisions have been loaded for the # subversion origin and get the number of the last one last_loaded_svn_rev = self.get_last_loaded_svn_rev(self.svn_url) # Then for stale repository, check if the last loaded revision in the archive # is different from the last revision on the remote subversion server. # Skip the dump of all revisions and the loading process if they are identical # to save some disk space and processing time. last_loaded_snp_and_rev = self._latest_snapshot_revision(self.origin.url) if last_loaded_snp_and_rev is not None: last_loaded_snp, last_loaded_rev = last_loaded_snp_and_rev self.svnrepo = self.svn_repo( self.origin.url, self.origin.url, self.temp_dir, self.max_content_size ) stale_repository = self.svnrepo.head_revision() == last_loaded_svn_rev if stale_repository and self.check_history_not_altered( last_loaded_svn_rev, last_loaded_rev ): self._snapshot = last_loaded_snp self._last_revision = last_loaded_rev self.done = True self.skip_post_load = True return # Then try to generate a dump file containing relevant svn revisions # to load, an exception will be thrown if something wrong happened dump_path = self.dump_svn_revisions(self.svn_url, last_loaded_svn_rev) # Finally, mount the dump and load the repository self.log.debug('Mounting dump file with "svnadmin load".') _, self.repo_path = init_svn_repo_from_dump( dump_path, prefix=TEMPORARY_DIR_PREFIX_PATTERN, suffix="-%s" % os.getpid(), root_dir=self.temp_dir, ) self.svn_url = "file://%s" % self.repo_path super().prepare() def cleanup(self): super().cleanup() if self.temp_dir and os.path.exists(self.temp_dir): shutil.rmtree(self.temp_dir) def visit_status(self): if self.truncated_dump: return "partial" else: return super().visit_status() diff --git a/swh/loader/svn/tasks.py b/swh/loader/svn/tasks.py index de72c68..4f60bfb 100644 --- a/swh/loader/svn/tasks.py +++ b/swh/loader/svn/tasks.py @@ -1,111 +1,46 @@ -# Copyright (C) 2015-2021 The Software Heritage developers +# Copyright (C) 2015-2022 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU General Public License version 3, or any later version # See top-level LICENSE file for more information -from datetime import datetime -from typing import Optional from celery import shared_task -import iso8601 + +from swh.loader.core.utils import parse_visit_date from .loader import SvnLoader, SvnLoaderFromDumpArchive, SvnLoaderFromRemoteDump -def convert_to_datetime(date: Optional[str]) -> Optional[datetime]: - if date is None: - return None - try: - assert isinstance(date, str) - return iso8601.parse_date(date) - except Exception: - return None +def _process_kwargs(kwargs): + if "visit_date" in kwargs: + kwargs["visit_date"] = parse_visit_date(kwargs["visit_date"]) + return kwargs @shared_task(name=__name__ + ".LoadSvnRepository") -def load_svn( - *, - url: Optional[str] = None, - origin_url: Optional[str] = None, - visit_date: Optional[str] = None, - incremental: Optional[bool] = True, -): - """Import a svn repository - - Args: - url: (mandatory) svn's repository url to ingest data from - origin_url: Optional original url override to use as origin reference in the - archive. If not provided, "url" is used as origin. - visit_date: Optional date to override the visit date - incremental: If True, the default, starts from the last snapshot (if any). - Otherwise, starts from the initial commit of the repository. - - - - """ - loader = SvnLoader.from_configfile( - url=url, - origin_url=origin_url, - visit_date=convert_to_datetime(visit_date), - incremental=incremental, - ) +def load_svn(**kwargs): + """Import a svn repository""" + loader = SvnLoader.from_configfile(**_process_kwargs(kwargs)) return loader.load() @shared_task(name=__name__ + ".MountAndLoadSvnRepository") -def load_svn_from_archive( - *, - url: Optional[str] = None, - archive_path: Optional[str] = None, - visit_date: Optional[str] = None, - incremental: Optional[bool] = True, -): - """1. Mount an svn dump from archive as a local svn repository - 2. Load it through the svn loader - 3. Clean up mounted svn repository archive - - Args: - url: origin url - archive_path: Path on disk to the archive holdin the svn repository to ingest - visit_date: Optional date to override the visit date - incremental: If True, the default, starts from the last snapshot (if any). - Otherwise, starts from the initial commit of the repository. - +def load_svn_from_archive(**kwargs): + """ + 1. Mount an svn dump from archive as a local svn repository + 2. Load it through the svn loader + 3. Clean up mounted svn repository archive """ - loader = SvnLoaderFromDumpArchive.from_configfile( - url=url, - archive_path=archive_path, - visit_date=convert_to_datetime(visit_date), - incremental=incremental, - ) + loader = SvnLoaderFromDumpArchive.from_configfile(**_process_kwargs(kwargs)) return loader.load() @shared_task(name=__name__ + ".DumpMountAndLoadSvnRepository") -def load_svn_from_remote_dump( - *, - url: Optional[str] = None, - origin_url: Optional[str] = None, - visit_date: Optional[str] = None, - incremental: Optional[bool] = True, -): - """1. Mount a remote svn dump as a local svn repository. - 2. Load it through the svn loader. - 3. Clean up mounted svn repository archive. - - Args: - url: (mandatory) svn's repository url to ingest data from - origin_url: Optional original url override to use as origin reference - in the archive. If not provided, "url" is used as origin. - visit_date: Optional date to override the visit date - incremental: If True, the default, starts from the last snapshot (if any). - Otherwise, starts from the initial commit of the repository. - +def load_svn_from_remote_dump(**kwargs): + """ + 1. Mount a remote svn dump as a local svn repository. + 2. Load it through the svn loader. + 3. Clean up mounted svn repository archive. """ - loader = SvnLoaderFromRemoteDump.from_configfile( - url=url, - origin_url=origin_url, - visit_date=convert_to_datetime(visit_date), - incremental=incremental, - ) + loader = SvnLoaderFromRemoteDump.from_configfile(**_process_kwargs(kwargs)) return loader.load() diff --git a/swh/loader/svn/tests/test_task.py b/swh/loader/svn/tests/test_task.py index ebdbe8c..efd88e0 100644 --- a/swh/loader/svn/tests/test_task.py +++ b/swh/loader/svn/tests/test_task.py @@ -1,79 +1,156 @@ -# Copyright (C) 2019-2021 The Software Heritage developers +# Copyright (C) 2019-2022 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU General Public License version 3, or any later version # See top-level LICENSE file for more information -from datetime import datetime, timezone +import uuid import pytest -from swh.loader.svn.tasks import convert_to_datetime +from swh.scheduler.model import ListedOrigin, Lister +from swh.scheduler.utils import create_origin_task_dict + + +@pytest.fixture(autouse=True) +def celery_worker_and_swh_config(swh_scheduler_celery_worker, swh_config): + pass + + +@pytest.fixture +def svn_lister(): + return Lister(name="svn-lister", instance_name="example", id=uuid.uuid4()) + + +@pytest.fixture +def svn_listed_origin(svn_lister): + return ListedOrigin( + lister_id=svn_lister.id, url="svn://example.org/repo", visit_type="svn" + ) + + +@pytest.fixture +def task_dict(svn_lister, svn_listed_origin): + return create_origin_task_dict(svn_listed_origin, svn_lister) def test_svn_loader( - mocker, swh_scheduler_celery_app, swh_scheduler_celery_worker, swh_config + mocker, + swh_scheduler_celery_app, ): mock_loader = mocker.patch("swh.loader.svn.loader.SvnLoader.load") mock_loader.return_value = {"status": "eventful"} res = swh_scheduler_celery_app.send_task( "swh.loader.svn.tasks.LoadSvnRepository", - kwargs=dict(url="some-technical-url", origin_url="origin-url"), + kwargs=dict( + url="some-technical-url", origin_url="origin-url", visit_date="now" + ), + ) + assert res + res.wait() + assert res.successful() + + assert res.result == {"status": "eventful"} + + +def test_svn_loader_for_listed_origin( + mocker, + swh_scheduler_celery_app, + task_dict, +): + mock_loader = mocker.patch("swh.loader.svn.loader.SvnLoader.load") + mock_loader.return_value = {"status": "eventful"} + + res = swh_scheduler_celery_app.send_task( + "swh.loader.svn.tasks.LoadSvnRepository", + args=task_dict["arguments"]["args"], + kwargs=task_dict["arguments"]["kwargs"], ) assert res res.wait() assert res.successful() assert res.result == {"status": "eventful"} def test_svn_loader_from_dump( - mocker, swh_scheduler_celery_app, swh_scheduler_celery_worker, swh_config + mocker, + swh_scheduler_celery_app, +): + mock_loader = mocker.patch("swh.loader.svn.loader.SvnLoaderFromDumpArchive.load") + mock_loader.return_value = {"status": "eventful"} + + res = swh_scheduler_celery_app.send_task( + "swh.loader.svn.tasks.MountAndLoadSvnRepository", + kwargs=dict(url="some-url", archive_path="some-path", visit_date="now"), + ) + assert res + res.wait() + assert res.successful() + + assert res.result == {"status": "eventful"} + + +def test_svn_loader_from_dump_for_listed_origin( + mocker, + swh_scheduler_celery_app, + svn_lister, + svn_listed_origin, ): mock_loader = mocker.patch("swh.loader.svn.loader.SvnLoaderFromDumpArchive.load") mock_loader.return_value = {"status": "eventful"} + svn_listed_origin.extra_loader_arguments = {"archive_path": "some-path"} + + task_dict = create_origin_task_dict(svn_listed_origin, svn_lister) + res = swh_scheduler_celery_app.send_task( "swh.loader.svn.tasks.MountAndLoadSvnRepository", - kwargs=dict(url="some-url", archive_path="some-path"), + args=task_dict["arguments"]["args"], + kwargs=task_dict["arguments"]["kwargs"], ) assert res res.wait() assert res.successful() assert res.result == {"status": "eventful"} def test_svn_loader_from_remote_dump( - mocker, swh_scheduler_celery_app, swh_scheduler_celery_worker, swh_config + mocker, + swh_scheduler_celery_app, ): mock_loader = mocker.patch("swh.loader.svn.loader.SvnLoaderFromRemoteDump.load") mock_loader.return_value = {"status": "eventful"} res = swh_scheduler_celery_app.send_task( "swh.loader.svn.tasks.DumpMountAndLoadSvnRepository", - kwargs=dict(url="some-remote-dump-url", origin_url="origin-url"), + kwargs=dict( + url="some-remote-dump-url", origin_url="origin-url", visit_date="now" + ), ) assert res res.wait() assert res.successful() assert res.result == {"status": "eventful"} -@pytest.mark.parametrize( - "date,expected_result", - [ - (None, None), - ( - "2021-11-23 09:41:02.434195+00:00", - datetime(2021, 11, 23, 9, 41, 2, 434195, tzinfo=timezone.utc), - ), - ( - "23112021", - None, - ), # failure to parse - ], -) -def test_convert_to_datetime(date, expected_result): - assert convert_to_datetime(date) == expected_result +def test_svn_loader_from_remote_dump_for_listed_origin( + mocker, + swh_scheduler_celery_app, + task_dict, +): + mock_loader = mocker.patch("swh.loader.svn.loader.SvnLoaderFromRemoteDump.load") + mock_loader.return_value = {"status": "eventful"} + + res = swh_scheduler_celery_app.send_task( + "swh.loader.svn.tasks.DumpMountAndLoadSvnRepository", + args=task_dict["arguments"]["args"], + kwargs=task_dict["arguments"]["kwargs"], + ) + assert res + res.wait() + assert res.successful() + + assert res.result == {"status": "eventful"} diff --git a/tox.ini b/tox.ini index b9438e6..3bcbc8f 100644 --- a/tox.ini +++ b/tox.ini @@ -1,75 +1,75 @@ [tox] envlist=black,flake8,mypy,py3 [testenv] extras = testing deps = pytest-cov swh.scheduler[testing] >= 0.5.0 dev: pdbpp commands = pytest --cov={envsitepackagesdir}/swh/loader/svn \ {envsitepackagesdir}/swh/loader/svn \ --cov-branch {posargs} [testenv:black] skip_install = true deps = black==22.3.0 commands = {envpython} -m black --check swh [testenv:flake8] skip_install = true deps = flake8==4.0.1 flake8-bugbear==22.3.23 commands = {envpython} -m flake8 [testenv:mypy] extras = testing deps = - mypy==0.920 + mypy==0.942 commands = mypy swh # build documentation outside swh-environment using the current # git HEAD of swh-docs, is executed on CI for each diff to prevent # breaking doc build [testenv:sphinx] whitelist_externals = make usedevelop = true extras = testing deps = # fetch and install swh-docs in develop mode -e git+https://forge.softwareheritage.org/source/swh-docs#egg=swh.docs setenv = SWH_PACKAGE_DOC_TOX_BUILD = 1 # turn warnings into errors SPHINXOPTS = -W commands = make -I ../.tox/sphinx/src/swh-docs/swh/ -C docs # build documentation only inside swh-environment using local state # of swh-docs package [testenv:sphinx-dev] whitelist_externals = make usedevelop = true extras = testing deps = # install swh-docs in develop mode -e ../swh-docs setenv = SWH_PACKAGE_DOC_TOX_BUILD = 1 # turn warnings into errors SPHINXOPTS = -W commands = make -I ../.tox/sphinx-dev/src/swh-docs/swh/ -C docs