diff --git a/PKG-INFO b/PKG-INFO index 0de803a..1462b4d 100644 --- a/PKG-INFO +++ b/PKG-INFO @@ -1,52 +1,52 @@ Metadata-Version: 2.1 Name: swh.loader.core -Version: 4.0.0 +Version: 4.1.0 Summary: Software Heritage Base Loader Home-page: https://forge.softwareheritage.org/diffusion/DLDBASE Author: Software Heritage developers Author-email: swh-devel@inria.fr Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest Project-URL: Funding, https://www.softwareheritage.org/donate Project-URL: Source, https://forge.softwareheritage.org/source/swh-loader-core Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-loader-core/ Classifier: Programming Language :: Python :: 3 Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Classifier: Operating System :: OS Independent Classifier: Development Status :: 5 - Production/Stable Requires-Python: >=3.7 Description-Content-Type: text/markdown Provides-Extra: testing License-File: LICENSE License-File: AUTHORS Software Heritage - Loader foundations ====================================== The Software Heritage Loader Core is a low-level loading utilities and helpers used by :term:`loaders `. The main entry points are classes: - :class:`swh.loader.core.loader.BaseLoader` for loaders (e.g. svn) - :class:`swh.loader.core.loader.DVCSLoader` for DVCS loaders (e.g. hg, git, ...) - :class:`swh.loader.package.loader.PackageLoader` for Package loaders (e.g. PyPI, Npm, ...) Package loaders --------------- This package also implements many package loaders directly, out of convenience, as they usually are quite similar and each fits in a single file. They all roughly follow these steps, explained in the :py:meth:`swh.loader.package.loader.PackageLoader.load` documentation. See the :ref:`package-loader-tutorial` for details. VCS loaders ----------- Unlike package loaders, VCS loaders remain in separate packages, as they often need more advanced conversions and very VCS-specific operations. This usually involves getting the branches of a repository and recursively loading revisions in the history (and directory trees in these revisions), until a known revision is found diff --git a/debian/changelog b/debian/changelog index e57bbe9..4bf56f5 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,1806 +1,1811 @@ -swh-loader-core (4.0.0-1~swh2~bpo10+1) buster-swh; urgency=medium +swh-loader-core (4.1.0-1~swh1) unstable-swh; urgency=medium - * Rebuild for buster-swh + * New upstream release 4.1.0 - (tagged by Antoine R. Dumont + (@ardumont) on 2022-09-14 09:36:57 + +0200) + * Upstream changes: - v4.1.0 - Pubdev: Ensure we correctly + parse and sort release versions - loader: Add origin URL and + visit type as sentry tags - -- Software Heritage autobuilder (on jenkins-debian1) Fri, 09 Sep 2022 09:50:49 +0000 + -- Software Heritage autobuilder (on jenkins-debian1) Wed, 14 Sep 2022 07:41:42 +0000 swh-loader-core (4.0.0-1~swh2) unstable-swh; urgency=medium * Update build dependencies and bump new release -- Antoine R. Dumont (@ardumont) Fri, 09 Sep 2022 11:47:53 +0200 swh-loader-core (4.0.0-1~swh1) unstable-swh; urgency=medium * New upstream release 4.0.0 - (tagged by Antoine R. Dumont (@ardumont) on 2022-09-09 09:03:40 +0200) * Upstream changes: - v4.0.0 - New package loader Golang - New package loader pubdev - New package loader Arch Linux - New package loader Arch Linux User - New package loader Crates - docs: Mention caveats of using archive checksums as ExtID. - package/utils: Add retry policy to download in case of throttling - package/archive: Handle tarball artifact with null time - Initialize 'status' before try block - Always log an error when setting 'failed' status - Add method process_data(), run between fetch_data() and store_data() -- Software Heritage autobuilder (on jenkins-debian1) Fri, 09 Sep 2022 07:11:14 +0000 swh-loader-core (3.5.0-1~swh1) unstable-swh; urgency=medium * New upstream release 3.5.0 - (tagged by Valentin Lorentz on 2022-05-20 14:19:02 +0200) * Upstream changes: - v3.5.0 - * BaseLoader.flush: Return the output of storage.flush -- Software Heritage autobuilder (on jenkins-debian1) Fri, 20 May 2022 12:24:23 +0000 swh-loader-core (3.4.1-1~swh1) unstable-swh; urgency=medium * New upstream release 3.4.1 - (tagged by Antoine R. Dumont (@ardumont) on 2022-05-13 10:22:15 +0200) * Upstream changes: - v3.4.1 - Initialize the success boolean early to avoid unbound exception -- Software Heritage autobuilder (on jenkins-debian1) Fri, 13 May 2022 08:26:59 +0000 swh-loader-core (3.4.0-1~swh1) unstable-swh; urgency=medium * New upstream release 3.4.0 - (tagged by Valentin Lorentz on 2022-05-06 10:36:18 +0200) * Upstream changes: - v3.4.0 - * crates: Do not literalinclude JSON file in ExtrinsicPackageMetadata doc - * Add Sentry Captures - * maven: Use most recent release of a package as default version - * loader.core: Add statsd timing and metadata metrics -- Software Heritage autobuilder (on jenkins-debian1) Fri, 06 May 2022 08:42:14 +0000 swh-loader-core (3.3.0-1~swh1) unstable-swh; urgency=medium * New upstream release 3.3.0 - (tagged by Antoine R. Dumont (@ardumont) on 2022-04-29 14:46:13 +0200) * Upstream changes: - v3.3.0 - Rust lang, Crates loader - package/maven: Fix jar archive download after changes in lister -- Software Heritage autobuilder (on jenkins-debian1) Fri, 29 Apr 2022 12:51:12 +0000 swh-loader-core (3.2.0-1~swh1) unstable-swh; urgency=medium * New upstream release 3.2.0 - (tagged by Valentin Lorentz on 2022-04-27 16:26:36 +0200) * Upstream changes: - v3.2.0 - * Store the result of MetadataFetcher.get_parent_origins - * cli: Pass metadata_fetcher_credentials from the config to the loader -- Software Heritage autobuilder (on jenkins-debian1) Wed, 27 Apr 2022 14:31:04 +0000 swh-loader-core (3.1.0-1~swh1) unstable-swh; urgency=medium * New upstream release 3.1.0 - (tagged by Valentin Lorentz on 2022-04-26 11:36:30 +0200) * Upstream changes: - v3.1.0 - * package loaders: Simplify initialization - * BaseLoader: Add hook to call metadata fetchers before loading an origin - * pre-commit maintenance - * debian: Fix loading when md5sum is missing in dsc file -- Software Heritage autobuilder (on jenkins-debian1) Tue, 26 Apr 2022 09:41:57 +0000 swh-loader-core (3.0.0-1~swh1) unstable-swh; urgency=medium * New upstream release 3.0.0 - (tagged by Valentin Lorentz on 2022-04-21 10:27:07 +0200) * Upstream changes: - v3.0.0 - * Remove unused function BaseLoader.store_metadata. - * Remove unused BaseLoader.origin_metadata attribute - * Replace self.url with self.origin.url in package loaders - * BaseLoader: Add 'origin_url' argument and remove 'prepare_origin_visit' method -- Software Heritage autobuilder (on jenkins-debian1) Thu, 21 Apr 2022 08:31:52 +0000 swh-loader-core (2.6.2-1~swh1) unstable-swh; urgency=medium * New upstream release 2.6.2 - (tagged by Antoine R. Dumont (@ardumont) on 2022-04-14 11:46:06 +0200) * Upstream changes: - v2.6.2 - maven: Consistently read lister input to ingest a mvn origin -- Software Heritage autobuilder (on jenkins-debian1) Thu, 14 Apr 2022 09:53:26 +0000 swh-loader-core (2.6.1-1~swh1) unstable-swh; urgency=medium * New upstream release 2.6.1 - (tagged by Antoine R. Dumont (@ardumont) on 2022-04-08 11:03:06 +0200) * Upstream changes: - v2.6.1 - Rename metadata key in data received from the deposit server - origin/master npm: Add all fields we use to the ExtID manifest - npm: Include package version id in ExtID manifest -- Software Heritage autobuilder (on jenkins-debian1) Fri, 08 Apr 2022 09:13:17 +0000 swh-loader-core (2.6.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.6.0 - (tagged by Valentin Lorentz on 2022-03-02 13:54:45 +0100) * Upstream changes: - v2.6.0 - * Update for the new output format of the Deposit's API. -- Software Heritage autobuilder (on jenkins-debian1) Wed, 02 Mar 2022 12:58:43 +0000 swh-loader-core (2.5.4-1~swh2) unstable-swh; urgency=medium * Bump new release with opam tests deactivated -- Antoine R. Dumont (@ardumont) Fri, 25 Feb 2022 12:40:40 +0100 swh-loader-core (2.5.4-1~swh1) unstable-swh; urgency=medium * New upstream release 2.5.4 - (tagged by Antoine R. Dumont (@ardumont) on 2022-02-25 10:23:51 +0100) * Upstream changes: - v2.5.4 - loader/opam/tests: Do not run actual opam init command call -- Software Heritage autobuilder (on jenkins-debian1) Fri, 25 Feb 2022 09:28:10 +0000 swh-loader-core (2.5.3-1~swh1) unstable-swh; urgency=medium * New upstream release 2.5.3 - (tagged by Antoine R. Dumont (@ardumont) on 2022-02-24 16:02:53 +0100) * Upstream changes: - v2.5.3 - opam: Allow build to run the opam init completely -- Software Heritage autobuilder (on jenkins-debian1) Thu, 24 Feb 2022 15:07:20 +0000 swh-loader-core (2.5.2-1~swh1) unstable-swh; urgency=medium * New upstream release 2.5.2 - (tagged by Valentin Lorentz on 2022-02-24 09:52:26 +0100) * Upstream changes: - v2.5.2 - * deposit: Remove unused raw_info -- Software Heritage autobuilder (on jenkins-debian1) Thu, 24 Feb 2022 08:57:52 +0000 swh-loader-core (2.5.1-1~swh1) unstable-swh; urgency=medium * New upstream release 2.5.1 - (tagged by Antoine R. Dumont (@ardumont) on 2022-02-16 15:27:02 +0100) * Upstream changes: - v2.5.1 - Add URL and directory to CLI loader status echo - Fix load_maven scheduling task name - docs: Fix typo detected with codespell - pre-commit: Bump hooks and add new one to check commit message spelling -- Software Heritage autobuilder (on jenkins-debian1) Wed, 16 Feb 2022 14:30:47 +0000 swh-loader-core (2.5.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.5.0 - (tagged by Antoine R. Dumont (@ardumont) on 2022-02-08 10:46:14 +0100) * Upstream changes: - v2.5.0 - Move visit date helper from hg loader to core -- Software Heritage autobuilder (on jenkins-debian1) Tue, 08 Feb 2022 09:49:53 +0000 swh-loader-core (2.4.1-1~swh1) unstable-swh; urgency=medium * New upstream release 2.4.1 - (tagged by Nicolas Dandrimont on 2022-02-03 14:12:05 +0100) * Upstream changes: - Release swh.loader.core 2.4.1 - fix Person mangling -- Software Heritage autobuilder (on jenkins-debian1) Thu, 03 Feb 2022 13:17:35 +0000 swh-loader-core (2.3.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.3.0 - (tagged by Nicolas Dandrimont on 2022-01-24 11:18:43 +0100) * Upstream changes: - Release swh.loader.core - Stop using the deprecated 'TimestampWithTimezone.offset' attribute - Include clone_with_timeout utility from swh.loader.mercurial -- Software Heritage autobuilder (on jenkins-debian1) Mon, 24 Jan 2022 10:22:35 +0000 swh-loader-core (2.2.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.2.0 - (tagged by Antoine R. Dumont (@ardumont) on 2022-01-18 14:33:08 +0100) * Upstream changes: - v2.2.0 - tests: Replace 'offset' and 'negative_utc' with 'offset_bytes' - deposit: Remove 'negative_utc' from test data - tests: Use TimestampWithTimezone.from_datetime() instead of the constructor - Add releases notes (from user-provided Atom document) to release messages. - deposit: Strip 'offset_bytes' from date dicts to support swh-model 4.0.0 - Pin mypy and drop type annotations which makes mypy unhappy -- Software Heritage autobuilder (on jenkins-debian1) Tue, 18 Jan 2022 15:52:53 +0000 swh-loader-core (2.1.1-1~swh1) unstable-swh; urgency=medium * New upstream release 2.1.1 - (tagged by Valentin Lorentz on 2021-12-09 17:14:12 +0100) * Upstream changes: - v2.1.1 - * nixguix: Fix crash when filtering extids on archives that were already loaded, but only from different URLs -- Software Heritage autobuilder (on jenkins-debian1) Thu, 09 Dec 2021 16:17:54 +0000 swh-loader-core (2.1.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.1.0 - (tagged by Valentin Lorentz on 2021-12-09 16:34:51 +0100) * Upstream changes: - v2.1.0 - * maven: various refactorings - * nixguix: Filter out releases with URLs different from the expected one -- Software Heritage autobuilder (on jenkins-debian1) Thu, 09 Dec 2021 15:38:14 +0000 swh-loader-core (2.0.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.0.0 - (tagged by Antoine R. Dumont (@ardumont) on 2021-12-07 15:53:23 +0100) * Upstream changes: - v2.0.0 - package-loaders: Add support for extid versions, and bump it for Debian - debian: Remove the extrinsic version from release names - debian: Fix confusion between the two versions -- Software Heritage autobuilder (on jenkins-debian1) Tue, 07 Dec 2021 14:57:19 +0000 swh-loader-core (1.3.0-1~swh1) unstable-swh; urgency=medium * New upstream release 1.3.0 - (tagged by Antoine Lambert on 2021-12-07 10:54:49 +0100) * Upstream changes: - version 1.3.0 -- Software Heritage autobuilder (on jenkins-debian1) Tue, 07 Dec 2021 09:58:53 +0000 swh-loader-core (1.2.1-1~swh1) unstable-swh; urgency=medium * New upstream release 1.2.1 - (tagged by Antoine R. Dumont (@ardumont) on 2021-12-03 16:15:32 +0100) * Upstream changes: - v1.2.1 - package.loader: Deduplicate extid target -- Software Heritage autobuilder (on jenkins-debian1) Fri, 03 Dec 2021 15:19:13 +0000 swh-loader-core (1.2.0-1~swh1) unstable-swh; urgency=medium * New upstream release 1.2.0 - (tagged by Antoine R. Dumont (@ardumont) on 2021-12-03 12:16:04 +0100) * Upstream changes: - v1.2.0 - debian: Rename loading task function to fix scheduling - debian: Handle extra sha1 sum in source package metadata - debian: Remove unused date parameter of DebianLoader - package.loader: Deduplicate target SWHIDs - package-loader-tutorial: Update to mention releases instead of revisions - package-loader-tutorial: Add a checklist - package-loader-tutorial: Highlight the recommendation to submit the loader early. -- Software Heritage autobuilder (on jenkins-debian1) Fri, 03 Dec 2021 11:19:52 +0000 swh-loader-core (1.1.0-1~swh1) unstable-swh; urgency=medium * New upstream release 1.1.0 - (tagged by Valentin Lorentz on 2021-11-22 11:58:11 +0100) * Upstream changes: - v1.1.0 - * Package loader: Uniformize author and message -- Software Heritage autobuilder (on jenkins-debian1) Mon, 22 Nov 2021 11:01:45 +0000 swh-loader-core (1.0.1-1~swh1) unstable-swh; urgency=medium * New upstream release 1.0.1 - (tagged by Valentin Lorentz on 2021-11-10 14:47:52 +0100) * Upstream changes: - v1.0.1 - * utils: Add types and let log instruction do the formatting - * Fix tests when run by gbp on Sid. -- Software Heritage autobuilder (on jenkins-debian1) Wed, 10 Nov 2021 13:53:43 +0000 swh-loader-core (1.0.0-1~swh1) unstable-swh; urgency=medium * New upstream release 1.0.0 - (tagged by Valentin Lorentz on 2021-11-10 14:25:24 +0100) * Upstream changes: - v1.0.0 - Main change: thismakes package loaders write releases instead of revisions - Other more-or-less related changes: - * Add missing documentation for `get_metadata_authority`. - * opam: Write package definitions to the extrinsic metadata storage - * deposit: Remove 'parent' deposit - * cleanup tests and unused code - * Document how each package loader populates fields. - * Refactor package loaders to make the version part of BasePackageInfo -- Software Heritage autobuilder (on jenkins-debian1) Wed, 10 Nov 2021 13:38:43 +0000 swh-loader-core (0.25.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.25.0 - (tagged by Antoine R. Dumont (@ardumont) on 2021-09-29 09:19:10 +0200) * Upstream changes: - v0.25.0 - Allow opam loader to actually use multi-instance opam root - opam: Define a initialize_opam_root parameter for opam loader -- Software Heritage autobuilder (on jenkins-debian1) Wed, 29 Sep 2021 07:26:12 +0000 swh-loader-core (0.23.5-1~swh1) unstable-swh; urgency=medium * New upstream release 0.23.5 - (tagged by Antoine R. Dumont (@ardumont) on 2021-09-24 17:31:22 +0200) * Upstream changes: - v0.23.5 - opam: Initialize opam root directory outside the constructor -- Software Heritage autobuilder (on jenkins-debian1) Fri, 24 Sep 2021 15:34:52 +0000 swh-loader-core (0.23.4-1~swh1) unstable-swh; urgency=medium * New upstream release 0.23.4 - (tagged by Antoine R. Dumont (@ardumont) on 2021-09-20 11:53:11 +0200) * Upstream changes: - v0.23.4 - Ensure that filename fallback out of an url is properly sanitized -- Software Heritage autobuilder (on jenkins-debian1) Mon, 20 Sep 2021 09:56:31 +0000 swh-loader-core (0.23.3-1~swh1) unstable-swh; urgency=medium * New upstream release 0.23.3 - (tagged by Antoine Lambert on 2021-09-16 10:47:40 +0200) * Upstream changes: - version 0.23.3 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 16 Sep 2021 08:51:47 +0000 swh-loader-core (0.23.2-1~swh1) unstable-swh; urgency=medium * New upstream release 0.23.2 - (tagged by Valentin Lorentz on 2021-08-12 12:22:44 +0200) * Upstream changes: - v0.23.2 - * deposit: Update status_detail on loader failure -- Software Heritage autobuilder (on jenkins-debian1) Thu, 12 Aug 2021 10:25:44 +0000 swh-loader-core (0.23.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.23.1 - (tagged by Antoine R. Dumont (@ardumont) on 2021-08-05 16:11:02 +0200) * Upstream changes: - v0.23.1 - Fix pypi upload issue. -- Software Heritage autobuilder (on jenkins-debian1) Thu, 05 Aug 2021 14:20:37 +0000 swh-loader-core (0.22.3-1~swh1) unstable-swh; urgency=medium * New upstream release 0.22.3 - (tagged by Valentin Lorentz on 2021-06-25 14:50:40 +0200) * Upstream changes: - v0.22.3 - * Use the postgresql class to instantiate storage in tests - * package-loader-tutorial: Add anchor so it can be referenced from swh-docs -- Software Heritage autobuilder (on jenkins-debian1) Fri, 25 Jun 2021 12:57:33 +0000 swh-loader-core (0.22.2-1~swh1) unstable-swh; urgency=medium * New upstream release 0.22.2 - (tagged by Antoine Lambert on 2021-06-10 16:11:30 +0200) * Upstream changes: - version 0.22.2 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 10 Jun 2021 14:19:06 +0000 swh-loader-core (0.22.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.22.1 - (tagged by Antoine Lambert on 2021-05-27 14:02:35 +0200) * Upstream changes: - version 0.22.1 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 27 May 2021 12:20:04 +0000 swh-loader-core (0.22.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.22.0 - (tagged by Valentin Lorentz on 2021-04-15 15:13:56 +0200) * Upstream changes: - v0.22.0 - Documentation: - * Document the big picture view of VCS and package loaders - * Add a package loader tutorial. - * Write an overview of how to write VCS loaders. - * Fix various Sphinx warnings - Package loaders: - * Add sha512 as a valid field in dsc metadata - * package loaders: Stop reading/writing Revision.metadata -- Software Heritage autobuilder (on jenkins-debian1) Thu, 15 Apr 2021 13:18:13 +0000 swh-loader-core (0.21.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.21.0 - (tagged by Valentin Lorentz on 2021-03-30 17:19:13 +0200) * Upstream changes: - v0.21.0 - * tests: recompute ids when evolving RawExtrinsicMetadata objects, to support swh-model 2.0.0 - * deposit.loader: Make archive.tar the default_filename - * debian: Make resolve_revision_from use the sha256 of the .dsc - * package.loader.*: unify package "cache"/deduplication using ExtIDs - * package.loader: Lookup packages from the ExtID storage - * package.loader: Write to the ExtID storage -- Software Heritage autobuilder (on jenkins-debian1) Tue, 30 Mar 2021 15:26:35 +0000 swh-loader-core (0.20.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.20.0 - (tagged by Valentin Lorentz on 2021-03-02 10:52:18 +0100) * Upstream changes: - v0.20.0 - * RawExtrinsicMetadata: update to use the API in swh-model 1.0.0 -- Software Heritage autobuilder (on jenkins-debian1) Tue, 02 Mar 2021 09:57:21 +0000 swh-loader-core (0.19.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.19.0 - (tagged by Antoine R. Dumont (@ardumont) on 2021-02-25 15:52:12 +0100) * Upstream changes: - v0.19.0 - deposit: Make deposit loader deal with tarball as well - deposit: Update deposit status when the load status is 'partial' - Make finalize_visit a method instead of nested function. -- Software Heritage autobuilder (on jenkins-debian1) Thu, 25 Feb 2021 14:55:54 +0000 swh-loader-core (0.18.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.18.1 - (tagged by Antoine R. Dumont (@ardumont) on 2021-02-19 18:02:58 +0100) * Upstream changes: - v0.18.1 - nixguix: Fix missing max_content_size constructor parameter -- Software Heritage autobuilder (on jenkins-debian1) Fri, 19 Feb 2021 17:06:33 +0000 swh-loader-core (0.18.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.18.0 - (tagged by Antoine R. Dumont (@ardumont) on 2021-02-17 13:13:24 +0100) * Upstream changes: - v0.18.0 - core.loader: Merge Loader into BaseLoader - Unify loader instantiation - nixguix: Ensure interaction with the origin url for edge case tests -- Software Heritage autobuilder (on jenkins-debian1) Wed, 17 Feb 2021 12:16:47 +0000 swh-loader-core (0.17.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.17.0 - (tagged by Antoine R. Dumont (@ardumont) on 2021-02-11 11:20:55 +0100) * Upstream changes: - v0.17.0 - package: Mark visit as not_found when relevant - package: Mark visit status as failed when relevant - core: Allow vcs loaders to deal with not_found status - core: Mark visit status as failed when relevant - loader: Make loader write the origin_visit_status' type -- Software Heritage autobuilder (on jenkins-debian1) Thu, 11 Feb 2021 10:23:42 +0000 swh-loader-core (0.16.0-1~swh2) unstable-swh; urgency=medium * Bump dependencies -- Antoine R. Dumont (@ardumont) Wed, 03 Feb 2021 14:25:26 +0100 swh-loader-core (0.16.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.16.0 - (tagged by Antoine R. Dumont (@ardumont) on 2021-02-03 14:14:01 +0100) * Upstream changes: - v0.16.0 - Adapt origin_get_latest_visit_status according to latest api change - Add a cli section in the doc - tox.ini: Add swh.core[testing] requirement - Small docstring improvements in the deposit loader code -- Software Heritage autobuilder (on jenkins-debian1) Wed, 03 Feb 2021 13:17:30 +0000 swh-loader-core (0.15.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.15.0 - (tagged by Nicolas Dandrimont on 2020-11-03 17:21:21 +0100) * Upstream changes: - Release swh-loader-core v0.15.0 - Attach raw extrinsic metadata to directories, not revisions - Handle a bunch of deprecation warnings: - explicit args in swh.objstorage get_objstorage - id -> target for raw extrinsic metadata objects - positional arguments for storage.raw_extrinsic_metadata_get -- Software Heritage autobuilder (on jenkins-debian1) Tue, 03 Nov 2020 16:26:20 +0000 swh-loader-core (0.14.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.14.0 - (tagged by Valentin Lorentz on 2020-10-16 18:23:28 +0200) * Upstream changes: - v0.14.0 - * npm: write metadata on revisions instead of snapshots. - * pypi: write metadata on revisions instead of snapshots. - * deposit.loader: Avoid unnecessary metadata json transformation -- Software Heritage autobuilder (on jenkins-debian1) Fri, 16 Oct 2020 16:26:14 +0000 swh-loader-core (0.13.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.13.1 - (tagged by Antoine R. Dumont (@ardumont) on 2020-10-02 16:54:05 +0200) * Upstream changes: - v0.13.1 - core.loader: Allow config parameter passing through constructor - tox.ini: pin black to the pre-commit version (19.10b0) to avoid flip-flops -- Software Heritage autobuilder (on jenkins-debian1) Fri, 02 Oct 2020 14:55:59 +0000 swh-loader-core (0.13.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.13.0 - (tagged by Antoine R. Dumont (@ardumont) on 2020-10-02 13:18:55 +0200) * Upstream changes: - v0.13.0 - package.loader: Migrate away from SWHConfig mixin - core.loader: Migrate away from SWHConfig mixin - Expose deposit configuration only within the deposit tests -- Software Heritage autobuilder (on jenkins-debian1) Fri, 02 Oct 2020 11:21:55 +0000 swh-loader-core (0.12.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.12.0 - (tagged by Antoine R. Dumont (@ardumont) on 2020-10-01 16:03:45 +0200) * Upstream changes: - v0.12.0 - deposit: Adapt loader to send extrinsic raw metadata to the metadata storage - core.loader: Log information about origin currently being ingested - Adapt cli declaration entrypoint to swh.core 0.3 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 01 Oct 2020 14:04:59 +0000 swh-loader-core (0.11.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.11.0 - (tagged by Antoine R. Dumont (@ardumont) on 2020-09-18 10:19:56 +0200) * Upstream changes: - v0.11.0 - loader: Stop materializing full lists of objects to be stored - tests.get_stats: Don't return a 'person' count - python: Reorder imports with isort - pre-commit: Add isort hook and configuration - pre-commit: Update flake8 hook configuration - cli: speedup the `swh` cli command startup time -- Software Heritage autobuilder (on jenkins-debian1) Fri, 18 Sep 2020 09:12:18 +0000 swh-loader-core (0.10.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.10.0 - (tagged by Antoine R. Dumont (@ardumont) on 2020-09-04 13:19:29 +0200) * Upstream changes: - v0.10.0 - loader: Adapt to latest storage revision_get change - origin/master Rename metadata format 'original-artifact-json' to 'original-artifacts-json'. - Tell pytest not to recurse in dotdirs. - package loader: Add the 'url' to the 'original_artifact' extrinsic metadata. - Write 'original_artifact' metadata to the extrinsic metadata storage. - Move parts of _load_revision to a new _load_directory method. - tests: Don't use naive datetimes. - package.loader: Split the warning message into multiple chunks - Replace calls to snapshot_get with snapshot_get_all_branches. -- Software Heritage autobuilder (on jenkins-debian1) Fri, 04 Sep 2020 11:28:09 +0000 swh-loader-core (0.9.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.9.1 - (tagged by Antoine R. Dumont (@ardumont) on 2020-08-08 14:47:52 +0200) * Upstream changes: - v0.9.1 - nixguix: Make the unsupported artifact extensions configurable - package.loader: Log a failure summary report at the end of the task -- Software Heritage autobuilder (on jenkins-debian1) Sat, 08 Aug 2020 12:51:33 +0000 swh-loader-core (0.9.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.9.0 - (tagged by Antoine R. Dumont (@ardumont) on 2020-08-07 22:57:14 +0200) * Upstream changes: - v0.9.0 - nixguix: Filter out unsupported artifact extensions - swh.loader.tests: Use snapshot_get_all_branches in check_snapshot - test_npm: Adapt content_get_metadata call to content_get - npm: Fix assertion to use the correct storage api -- Software Heritage autobuilder (on jenkins-debian1) Fri, 07 Aug 2020 21:00:40 +0000 swh-loader-core (0.8.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.8.1 - (tagged by Antoine R. Dumont (@ardumont) on 2020-08-06 16:48:38 +0200) * Upstream changes: - v0.8.1 - Adapt code according to storage signature -- Software Heritage autobuilder (on jenkins-debian1) Thu, 06 Aug 2020 14:50:39 +0000 swh-loader-core (0.8.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.8.0 - (tagged by Antoine R. Dumont (@ardumont) on 2020-08-05 10:16:36 +0200) * Upstream changes: - v0.8.0 - archive: fix docstring - nixguix: Fix docstring - nixguix: Align error message formatting using f-string - nixguix: Fix format issue in error message - Convert the 'metadata' and 'info' cached-properties/lazy-attributes into methods - cran: fix call to logger.warning - pypi: Load the content of the API's response as extrinsic snapshot metadata - Add a default value for RawExtrinsicMetadataCore.discovery_date - npm: Load the content of the API's response as extrinsic snapshot metadata - Make retrieve_sources use generic api_info instead of duplicating its code - nixguix: Load the content of sources.json as extrinsic snapshot metadata - Update tests to accept PagedResult from storage.raw_extrinsic_metadata_get -- Software Heritage autobuilder (on jenkins-debian1) Wed, 05 Aug 2020 08:19:20 +0000 swh-loader-core (0.7.3-1~swh1) unstable-swh; urgency=medium * New upstream release 0.7.3 - (tagged by Valentin Lorentz on 2020-07-30 19:16:21 +0200) * Upstream changes: - v0.7.3 - core.loader: Fix Iterable/List typing issues - package.loader: Fix type warning -- Software Heritage autobuilder (on jenkins-debian1) Thu, 30 Jul 2020 17:23:57 +0000 swh-loader-core (0.7.2-1~swh1) unstable-swh; urgency=medium * New upstream release 0.7.2 - (tagged by Valentin Lorentz on 2020-07-29 11:41:39 +0200) * Upstream changes: - v0.7.2 - * Fix typo in message logged on extrinsic metadata loading errors. - * Don't pass non-sequence iterables to the storage API. -- Software Heritage autobuilder (on jenkins-debian1) Wed, 29 Jul 2020 09:45:52 +0000 swh-loader-core (0.7.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.7.1 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-28 12:14:02 +0200) * Upstream changes: - v0.7.1 - Apply rename of object_metadata to raw_extrinsic_metadata. -- Software Heritage autobuilder (on jenkins-debian1) Tue, 28 Jul 2020 10:16:56 +0000 swh-loader-core (0.6.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.6.1 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-23 11:12:29 +0200) * Upstream changes: - v0.6.1 - npm.loader: Fix null author parsing corner case - npm.loader: Fix author parsing corner case - npm.loader: Extract _author_str function + add types, tests - core.loader: docs: Update origin_add reference -- Software Heritage autobuilder (on jenkins-debian1) Thu, 23 Jul 2020 09:15:41 +0000 swh-loader-core (0.6.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.6.0 - (tagged by Valentin Lorentz on 2020-07-20 13:23:22 +0200) * Upstream changes: - v0.6.0 - * Use the new object_metadata_add endpoint instead of origin_metadata_add. - * Apply renaming of MetadataAuthorityType.DEPOSIT to MetadataAuthorityType.DEPOSIT_CLIENT. -- Software Heritage autobuilder (on jenkins-debian1) Mon, 20 Jul 2020 11:27:53 +0000 swh-loader-core (0.5.10-1~swh1) unstable-swh; urgency=medium * New upstream release 0.5.10 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-17 15:10:42 +0200) * Upstream changes: - v0.5.10 - test_init: Decrease assertion checks so debian package builds fine - test_nixguix: Simplify the nixguix specific check_snapshot function -- Software Heritage autobuilder (on jenkins-debian1) Fri, 17 Jul 2020 13:13:19 +0000 swh-loader-core (0.5.9-1~swh1) unstable-swh; urgency=medium * New upstream release 0.5.9 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-17 11:52:38 +0200) * Upstream changes: - v0.5.9 - test.check_snapshot: Drop accepting using dict for snapshot comparison - test: Check against snapshot model object -- Software Heritage autobuilder (on jenkins-debian1) Fri, 17 Jul 2020 09:55:12 +0000 swh-loader-core (0.5.8-1~swh1) unstable-swh; urgency=medium * New upstream release 0.5.8 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-16 17:18:17 +0200) * Upstream changes: - v0.5.8 - test_init: Use snapshot object -- Software Heritage autobuilder (on jenkins-debian1) Thu, 16 Jul 2020 15:20:49 +0000 swh-loader-core (0.5.7-1~swh1) unstable-swh; urgency=medium * New upstream release 0.5.7 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-16 16:10:57 +0200) * Upstream changes: - v0.5.7 - test_init: Fix tests using the latest swh-storage fixture -- Software Heritage autobuilder (on jenkins-debian1) Thu, 16 Jul 2020 14:14:59 +0000 swh-loader-core (0.5.5-1~swh1) unstable-swh; urgency=medium * New upstream release 0.5.5 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-15 12:34:09 +0200) * Upstream changes: - v0.5.5 - check_snapshot: Check existence down to contents - Expose a pytest_plugin module so other loaders can reuse for tests - pytest: Remove no longer needed pytest setup - Fix branches types in tests - Small code improvement in package/loader.py -- Software Heritage autobuilder (on jenkins-debian1) Wed, 15 Jul 2020 10:37:11 +0000 swh-loader-core (0.5.4-1~swh1) unstable-swh; urgency=medium * New upstream release 0.5.4 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-10 09:52:21 +0200) * Upstream changes: - v0.5.4 - Clean up the swh.scheduler / swh.storage pytest plugin imports -- Software Heritage autobuilder (on jenkins-debian1) Fri, 10 Jul 2020 07:54:56 +0000 swh-loader-core (0.5.3-1~swh1) unstable-swh; urgency=medium * New upstream release 0.5.3 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-09 09:46:21 +0200) * Upstream changes: - v0.5.3 - Update the revision metadata field as an immutable dict - tests: Use dedicated storage and scheduler fixtures - loaders.tests: Simplify and add coverage to check_snapshot -- Software Heritage autobuilder (on jenkins-debian1) Thu, 09 Jul 2020 07:48:33 +0000 swh-loader-core (0.5.2-1~swh1) unstable-swh; urgency=medium * New upstream release 0.5.2 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-07 12:29:17 +0200) * Upstream changes: - v0.5.2 - nixguix/loader: Check further the source entry only if it's valid - nixguix/loader: Allow version both as string or integer - Move remaining common test utility functions to top-level arborescence - Move common test utility function to the top-level arborescence - Define common test helper function - Reuse swh.model.from_disk.iter_directory function -- Software Heritage autobuilder (on jenkins-debian1) Tue, 07 Jul 2020 10:31:36 +0000 swh-loader-core (0.5.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.5.1 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-01 12:32:54 +0200) * Upstream changes: - v0.5.1 - Use origin_add instead of deprecated origin_add_one endpoint - Migrate to use object's "object_type" field when computing objects -- Software Heritage autobuilder (on jenkins-debian1) Wed, 01 Jul 2020 10:34:59 +0000 swh-loader-core (0.5.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.5.0 - (tagged by Antoine R. Dumont (@ardumont) on 2020-06-29 13:18:41 +0200) * Upstream changes: - v0.5.0 - loader*: Drop obsolete origin visit fields -- Software Heritage autobuilder (on jenkins-debian1) Mon, 29 Jun 2020 11:20:59 +0000 swh-loader-core (0.4.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.4.0 - (tagged by Antoine R. Dumont (@ardumont) on 2020-06-23 15:02:20 +0200) * Upstream changes: - v0.4.0 - loader: Retrieve latest snapshot with snapshot-get-latest function -- Software Heritage autobuilder (on jenkins-debian1) Tue, 23 Jun 2020 13:14:09 +0000 swh-loader-core (0.3.2-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.2 - (tagged by Antoine R. Dumont (@ardumont) on 2020-06-22 15:13:05 +0200) * Upstream changes: - v0.3.2 - Add helper function to ensure loader visit are as expected -- Software Heritage autobuilder (on jenkins-debian1) Mon, 22 Jun 2020 13:15:41 +0000 swh-loader-core (0.3.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.1 - (tagged by Antoine Lambert on 2020-06-12 16:43:18 +0200) * Upstream changes: - version 0.3.1 -- Software Heritage autobuilder (on jenkins-debian1) Fri, 12 Jun 2020 14:47:42 +0000 swh-loader-core (0.3.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.0 - (tagged by Antoine R. Dumont (@ardumont) on 2020-06-12 11:05:41 +0200) * Upstream changes: - v0.3.0 - Migrate to new storage.origin_visit_add endpoint - loader: Migrate to origin visit status - test_deposits: Fix origin_metadata_get which is a paginated endpoint - Fix a potential UnboundLocalError in clean_dangling_folders() -- Software Heritage autobuilder (on jenkins-debian1) Fri, 12 Jun 2020 09:08:17 +0000 swh-loader-core (0.2.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.2.0 - (tagged by David Douard on 2020-06-04 14:20:08 +0200) * Upstream changes: - v0.2.0 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 04 Jun 2020 12:25:57 +0000 swh-loader-core (0.1.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.1.0 - (tagged by Nicolas Dandrimont on 2020-05-29 16:01:11 +0200) * Upstream changes: - Release swh.loader.core v0.1.0 - Make sure partial visits don't reference unloaded snapshots - Ensure proper behavior when loading into partial archives (e.g. staging) - Improve test coverage -- Software Heritage autobuilder (on jenkins-debian1) Fri, 29 May 2020 14:05:36 +0000 swh-loader-core (0.0.97-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.97 - (tagged by Antoine R. Dumont (@ardumont) on 2020-05-26 14:22:51 +0200) * Upstream changes: - v0.0.97 - nixguix: catch and log artifact resolution failures - nixguix: Override known_artifacts to filter out "evaluation" branch - nixguix.tests: Add missing __init__ file -- Software Heritage autobuilder (on jenkins-debian1) Tue, 26 May 2020 12:25:35 +0000 swh-loader-core (0.0.96-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.96 - (tagged by Valentin Lorentz on 2020-05-19 18:42:23 +0200) * Upstream changes: - v0.0.96 - * Pass bytes instead a dict to origin_metadata_add. -- Software Heritage autobuilder (on jenkins-debian1) Tue, 19 May 2020 16:45:03 +0000 swh-loader-core (0.0.95-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.95 - (tagged by Valentin Lorentz on 2020-05-19 14:44:01 +0200) * Upstream changes: - v0.0.95 - * Use the new swh-storage API for storing metadata. -- Software Heritage autobuilder (on jenkins-debian1) Tue, 19 May 2020 12:47:48 +0000 swh-loader-core (0.0.94-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.94 - (tagged by Antoine R. Dumont (@ardumont) on 2020-05-15 12:49:22 +0200) * Upstream changes: - v0.0.94 - deposit: Adapt loader to use the latest deposit update api - tests: Use proper date initialization - setup.py: add documentation link -- Software Heritage autobuilder (on jenkins-debian1) Fri, 15 May 2020 10:52:16 +0000 swh-loader-core (0.0.93-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.93 - (tagged by Antoine R. Dumont (@ardumont) on 2020-04-23 16:43:16 +0200) * Upstream changes: - v0.0.93 - deposit.loader: Build revision out of the deposit api read metadata -- Software Heritage autobuilder (on jenkins-debian1) Thu, 23 Apr 2020 14:46:48 +0000 swh-loader-core (0.0.92-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.92 - (tagged by Antoine R. Dumont (@ardumont) on 2020-04-23 11:49:30 +0200) * Upstream changes: - v0.0.92 - deposit.loader: Fix revision metadata redundancy in deposit metadata - loader.deposit: Clarify FIXME intent - test_nixguix: Remove the incorrect fixme - test_nixguix: Add a fixme note on test_loader_two_visits - package.nixguix: Ensure the revisions are structurally sound -- Software Heritage autobuilder (on jenkins-debian1) Thu, 23 Apr 2020 09:52:18 +0000 swh-loader-core (0.0.91-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.91 - (tagged by Antoine R. Dumont (@ardumont) on 2020-04-21 15:59:55 +0200) * Upstream changes: - v0.0.91 - deposit.loader: Fix committer date appropriately - tests_deposit: Define specific requests_mock_datadir fixture - nixguix: Move helper function below the class definition - setup: Update the minimum required runtime python3 version -- Software Heritage autobuilder (on jenkins-debian1) Tue, 21 Apr 2020 14:02:51 +0000 swh-loader-core (0.0.90-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.90 - (tagged by Antoine R. Dumont (@ardumont) on 2020-04-15 14:27:01 +0200) * Upstream changes: - v0.0.90 - Improve exception handling -- Software Heritage autobuilder (on jenkins-debian1) Wed, 15 Apr 2020 12:30:07 +0000 swh-loader-core (0.0.89-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.89 - (tagged by Antoine R. Dumont (@ardumont) on 2020-04-14 15:48:15 +0200) * Upstream changes: - v0.0.89 - package.utils: Define a timeout on download connections - package.loader: Clear proxy buffer state when failing to load revision - Fix a couple of storage args deprecation warnings - cli: Sort loaders list and fix some tests - Add a pyproject.toml file to target py37 for black - Enable black -- Software Heritage autobuilder (on jenkins-debian1) Tue, 14 Apr 2020 15:30:08 +0000 swh-loader-core (0.0.88-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.88 - (tagged by Antoine R. Dumont (@ardumont) on 2020-04-03 15:52:07 +0200) * Upstream changes: - v0.0.88 - v0.0.88 nixguix: validate and clean sources.json structure -- Software Heritage autobuilder (on jenkins-debian1) Fri, 03 Apr 2020 13:54:24 +0000 swh-loader-core (0.0.87-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.87 - (tagged by Antoine R. Dumont (@ardumont) on 2020-04-02 14:37:37 +0200) * Upstream changes: - v0.0.87 - nixguix: rename the `url` source attribute to `urls` - nixguix: rename the test file - nixguix: add the integrity attribute in release metadata -- Software Heritage autobuilder (on jenkins-debian1) Thu, 02 Apr 2020 12:39:58 +0000 swh-loader-core (0.0.86-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.86 - (tagged by Antoine R. Dumont (@ardumont) on 2020-03-26 16:15:24 +0100) * Upstream changes: - v0.0.86 - core.loader: Remove origin_visit_update call from DVCSLoader class -- Software Heritage autobuilder (on jenkins-debian1) Thu, 26 Mar 2020 15:19:29 +0000 swh-loader-core (0.0.85-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.85 - (tagged by Antoine R. Dumont (@ardumont) on 2020-03-26 15:36:58 +0100) * Upstream changes: - v0.0.85 - core.loader: Allow core loader to update origin_visit in one call - Rename the functional loader to nixguix loader -- Software Heritage autobuilder (on jenkins-debian1) Thu, 26 Mar 2020 14:43:17 +0000 swh-loader-core (0.0.84-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.84 - (tagged by Antoine R. Dumont (@ardumont) on 2020-03-24 11:29:49 +0100) * Upstream changes: - v0.0.84 - test: Use storage endpoint to check latest origin visit status - package.loader: Fix status visit to 'partial' - package.loader: add a test to reproduce EOFError error -- Software Heritage autobuilder (on jenkins-debian1) Tue, 24 Mar 2020 10:32:55 +0000 swh-loader-core (0.0.83-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.83 - (tagged by Antoine R. Dumont (@ardumont) on 2020-03-23 15:16:14 +0100) * Upstream changes: - v0.0.83 - Make the swh.loader.package exception handling more granular - package.loader: Reference a snapshot on partial visit - package.loader: Extract a _load_snapshot method - functional: create a branch named evaluation pointing to the evaluation commit - package.loader: add extra_branches method -- Software Heritage autobuilder (on jenkins-debian1) Mon, 23 Mar 2020 14:19:43 +0000 swh-loader-core (0.0.82-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.82 - (tagged by Antoine R. Dumont (@ardumont) on 2020-03-18 11:55:48 +0100) * Upstream changes: - v0.0.82 - functional.loader: Add loader - package.loader: ignore non tarball source -- Software Heritage autobuilder (on jenkins-debian1) Wed, 18 Mar 2020 10:59:38 +0000 swh-loader-core (0.0.81-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.81 - (tagged by Antoine R. Dumont (@ardumont) on 2020-03-16 13:14:33 +0100) * Upstream changes: - v0.0.81 - Migrate to latest storage.origin_visit_add api change - Move Person parsing to swh- model. -- Software Heritage autobuilder (on jenkins-debian1) Mon, 16 Mar 2020 12:17:43 +0000 swh-loader-core (0.0.80-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.80 - (tagged by Valentin Lorentz on 2020-02-28 17:05:14 +0100) * Upstream changes: - v0.0.80 - * use swh-model objects instead of dicts. -- Software Heritage autobuilder (on jenkins-debian1) Fri, 28 Feb 2020 16:10:06 +0000 swh-loader-core (0.0.79-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.79 - (tagged by Antoine R. Dumont (@ardumont) on 2020-02-25 11:40:05 +0100) * Upstream changes: - v0.0.79 - Move revision loading logic to its own function. - Use swh-storage validation proxy earlier in the pipeline. - Use swh-storage validation proxy. - Add missing __init__.py and fix tests. -- Software Heritage autobuilder (on jenkins-debian1) Tue, 25 Feb 2020 10:48:07 +0000 swh-loader-core (0.0.78-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.78 - (tagged by Antoine R. Dumont (@ardumont) on 2020-02-06 15:28:11 +0100) * Upstream changes: - v0.0.78 - tests: Use new get_storage signature - loader.core.converters: Prefer the with open pattern to read file - test_converters: Add coverage on prepare_contents method - test_converters: Migrate to pytest - loader.core/package: Call storage's (skipped_)content_add endpoints -- Software Heritage autobuilder (on jenkins-debian1) Thu, 06 Feb 2020 15:09:05 +0000 swh-loader-core (0.0.77-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.77 - (tagged by Antoine R. Dumont (@ardumont) on 2020-01-30 10:32:08 +0100) * Upstream changes: - v0.0.77 - loader.npm: If no upload time provided, use artifact's mtime if provided - loader.npm: Fail ingestion if at least 1 artifact has no upload time -- Software Heritage autobuilder (on jenkins-debian1) Thu, 30 Jan 2020 09:37:58 +0000 swh-loader-core (0.0.76-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.76 - (tagged by Antoine R. Dumont (@ardumont) on 2020-01-28 13:07:30 +0100) * Upstream changes: - v0.0.76 - npm.loader: Skip artifacts with no intrinsic metadata - pypi.loader: Skip artifacts with no intrinsic metadata - package.loader: Fix edge case when some listing returns no content - core.loader: Drop retro- compatibility class names - loader.tests: Add filter and buffer proxy storage - docs: Fix sphinx warnings - README: Update class names -- Software Heritage autobuilder (on jenkins-debian1) Tue, 28 Jan 2020 12:11:07 +0000 swh-loader-core (0.0.75-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.75 - (tagged by Antoine R. Dumont (@ardumont) on 2020-01-16 14:14:29 +0100) * Upstream changes: - v0.0.75 - cran.loader: Align cran loader with other package loaders -- Software Heritage autobuilder (on jenkins-debian1) Thu, 16 Jan 2020 13:17:30 +0000 swh-loader-core (0.0.74-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.74 - (tagged by Antoine R. Dumont (@ardumont) on 2020-01-15 15:30:13 +0100) * Upstream changes: - v0.0.74 - Drop no longer used retrying dependency - core.loader: Clean up indirection and retry behavior - tests: Use retry proxy storage in loaders - core.loader: Drop dead code - cran.loader: Fix parsing description file error -- Software Heritage autobuilder (on jenkins-debian1) Wed, 15 Jan 2020 14:33:57 +0000 swh-loader-core (0.0.73-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.73 - (tagged by Antoine R. Dumont (@ardumont) on 2020-01-09 10:00:21 +0100) * Upstream changes: - v0.0.73 - package.cran: Name CRAN task appropriately -- Software Heritage autobuilder (on jenkins-debian1) Thu, 09 Jan 2020 09:05:07 +0000 swh-loader-core (0.0.72-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.72 - (tagged by Antoine R. Dumont (@ardumont) on 2020-01-06 16:37:58 +0100) * Upstream changes: - v0.0.72 - package.loader: Fail fast when unable to create origin/origin_visit - cran.loader: Add implementation -- Software Heritage autobuilder (on jenkins-debian1) Mon, 06 Jan 2020 15:50:08 +0000 swh-loader-core (0.0.71-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.71 - (tagged by Antoine R. Dumont (@ardumont) on 2019-12-20 14:22:31 +0100) * Upstream changes: - v0.0.71 - package.utils: Drop unneeded hashes from download computation -- Software Heritage autobuilder (on jenkins-debian1) Fri, 20 Dec 2019 13:26:09 +0000 swh-loader-core (0.0.70-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.70 - (tagged by Antoine R. Dumont (@ardumont) on 2019-12-20 11:32:09 +0100) * Upstream changes: - v0.0.70 - debian.loader: Improve and fix revision resolution's corner cases -- Software Heritage autobuilder (on jenkins-debian1) Fri, 20 Dec 2019 10:39:34 +0000 swh-loader-core (0.0.69-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.69 - (tagged by Antoine R. Dumont (@ardumont) on 2019-12-12 16:21:59 +0100) * Upstream changes: - v0.0.69 - loader.core: Fix correctly loader initialization -- Software Heritage autobuilder (on jenkins-debian1) Thu, 12 Dec 2019 15:26:13 +0000 swh-loader-core (0.0.68-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.68 - (tagged by Antoine R. Dumont (@ardumont) on 2019-12-12 15:45:21 +0100) * Upstream changes: - v0.0.68 - loader.core: Fix initialization issue in dvcs loaders -- Software Heritage autobuilder (on jenkins-debian1) Thu, 12 Dec 2019 14:49:12 +0000 swh-loader-core (0.0.67-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.67 - (tagged by Antoine R. Dumont (@ardumont) on 2019-12-12 14:02:47 +0100) * Upstream changes: - v0.0.67 - loader.core: Type methods - loader.core: Transform data input into list - loader.core: Add missing conversion step on content -- Software Heritage autobuilder (on jenkins-debian1) Thu, 12 Dec 2019 13:07:47 +0000 swh-loader-core (0.0.66-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.66 - (tagged by Antoine R. Dumont (@ardumont) on 2019-12-12 12:01:14 +0100) * Upstream changes: - v0.0.66 - Drop deprecated behavior -- Software Heritage autobuilder (on jenkins-debian1) Thu, 12 Dec 2019 11:05:17 +0000 swh-loader-core (0.0.65-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.65 - (tagged by Antoine R. Dumont (@ardumont) on 2019-12-12 11:42:46 +0100) * Upstream changes: - v0.0.65 - loader.cli: Improve current implementation - tasks: Enforce kwargs use in task message -- Software Heritage autobuilder (on jenkins-debian1) Thu, 12 Dec 2019 10:51:02 +0000 swh-loader-core (0.0.64-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.64 - (tagged by Antoine R. Dumont (@ardumont) on 2019-12-10 09:49:06 +0100) * Upstream changes: - v0.0.64 - requirements-test: Add missing test dependency - tests: Refactor using pytest-mock's mocker fixture - loader.cli: Add tests around cli - package.npm: Align loader instantiation - loader.cli: Reference new loader cli -- Software Heritage autobuilder (on jenkins-debian1) Tue, 10 Dec 2019 08:56:02 +0000 swh-loader-core (0.0.63-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.63 - (tagged by Antoine R. Dumont (@ardumont) on 2019-12-05 16:01:49 +0100) * Upstream changes: - v0.0.63 - Add missing inclusion instruction -- Software Heritage autobuilder (on jenkins-debian1) Thu, 05 Dec 2019 15:05:39 +0000 swh-loader-core (0.0.62-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.62 - (tagged by Antoine R. Dumont (@ardumont) on 2019-12-05 15:46:46 +0100) * Upstream changes: - v0.0.62 - Move package loaders to their own namespace -- Software Heritage autobuilder (on jenkins-debian1) Thu, 05 Dec 2019 14:50:19 +0000 swh-loader-core (0.0.61-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.61 - (tagged by Antoine R. Dumont (@ardumont) on 2019-11-28 17:25:49 +0100) * Upstream changes: - v0.0.61 - pypi: metadata -> revision: Deal with previous metadata format - npm: metadata -> revision: Deal with previous metadata format -- Software Heritage autobuilder (on jenkins-debian1) Thu, 28 Nov 2019 16:29:47 +0000 swh-loader-core (0.0.60-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.60 - (tagged by Antoine R. Dumont (@ardumont) on 2019-11-26 12:09:28 +0100) * Upstream changes: - v0.0.60 - package.deposit: Fix revision- get inconsistency - package.deposit: Provide parents in any case - package.deposit: Fix url computation issue - utils: Work around header issue during download -- Software Heritage autobuilder (on jenkins-debian1) Tue, 26 Nov 2019 11:18:41 +0000 swh-loader-core (0.0.59-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.59 - (tagged by Antoine R. Dumont (@ardumont) on 2019-11-22 18:11:33 +0100) * Upstream changes: - v0.0.59 - npm: Explicitly retrieve the revision date from extrinsic metadata -- Software Heritage autobuilder (on jenkins-debian1) Fri, 22 Nov 2019 17:15:34 +0000 swh-loader-core (0.0.58-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.58 - (tagged by Antoine R. Dumont (@ardumont) on 2019-11-22 12:08:10 +0100) * Upstream changes: - v0.0.58 - package.pypi: Filter out non- sdist package type -- Software Heritage autobuilder (on jenkins-debian1) Fri, 22 Nov 2019 11:11:56 +0000 swh-loader-core (0.0.57-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.57 - (tagged by Antoine R. Dumont (@ardumont) on 2019-11-22 11:26:11 +0100) * Upstream changes: - v0.0.57 - package.pypi: Fix project url computation edge case - Use pkg_resources to get the package version instead of vcversioner -- Software Heritage autobuilder (on jenkins-debian1) Fri, 22 Nov 2019 10:31:11 +0000 swh-loader-core (0.0.56-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.56 - (tagged by Antoine R. Dumont (@ardumont) on 2019-11-21 16:12:46 +0100) * Upstream changes: - v0.0.56 - package.tasks: Rename appropriately load_deb_package task type name - Fix typos reported by codespell - Add a pre-commit config file -- Software Heritage autobuilder (on jenkins-debian1) Thu, 21 Nov 2019 15:16:23 +0000 swh-loader-core (0.0.55-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.55 - (tagged by Antoine R. Dumont (@ardumont) on 2019-11-21 13:51:03 +0100) * Upstream changes: - v0.0.55 - package.tasks: Rename load_archive into load_archive_files - Migrate tox.ini to extras = xxx instead of deps = .[testing] - Merge tox test environments -- Software Heritage autobuilder (on jenkins-debian1) Thu, 21 Nov 2019 12:56:07 +0000 swh-loader-core (0.0.54-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.54 - (tagged by Antoine R. Dumont (@ardumont) on 2019-11-21 11:29:20 +0100) * Upstream changes: - v0.0.54 - loader.package.deposit: Drop swh.deposit.client requirement - Include all requirements in MANIFEST.in -- Software Heritage autobuilder (on jenkins-debian1) Thu, 21 Nov 2019 10:32:23 +0000 swh-loader-core (0.0.53-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.53 - (tagged by Antoine R. Dumont (@ardumont) on 2019-11-20 14:26:36 +0100) * Upstream changes: - v0.0.53 - loader.package.tasks: Document tasks - Define correctly the setup.py's entry_points -- Software Heritage autobuilder (on jenkins-debian1) Wed, 20 Nov 2019 13:30:10 +0000 swh-loader-core (0.0.52-1~swh3) unstable-swh; urgency=medium * Update dh-python version constraint -- Antoine R. Dumont (@ardumont) Wed, 20 Nov 2019 12:03:00 +0100 swh-loader-core (0.0.52-1~swh2) unstable-swh; urgency=medium * Add egg-info to pybuild.testfiles. -- Antoine R. Dumont (@ardumont) Wed, 20 Nov 2019 11:42:42 +0100 swh-loader-core (0.0.52-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.52 - (tagged by Antoine R. Dumont (@ardumont) on 2019-11-19 15:15:40 +0100) * Upstream changes: - v0.0.52 - Ensure BufferedLoader and UnbufferedLoader do flush their storage - loader.package: Register loader package tasks - package.tasks: Rename debian task to load_deb -- Software Heritage autobuilder (on jenkins-debian1) Tue, 19 Nov 2019 14:18:41 +0000 swh-loader-core (0.0.51-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.51 - (tagged by David Douard on 2019-11-18 17:05:17 +0100) * Upstream changes: - v0.0.51 -- Software Heritage autobuilder (on jenkins-debian1) Mon, 18 Nov 2019 16:09:44 +0000 swh-loader-core (0.0.50-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.50 - (tagged by Antoine R. Dumont (@ardumont) on 2019-11-13 15:56:55 +0100) * Upstream changes: - v0.0.50 - package.loader: Check snapshot_id is set as returned value - package.loader: Ensure the origin visit type is set appropriately - package.loader: Fix serialization issue - package.debian: Align origin_visit type to 'deb' as in production -- Software Heritage autobuilder (on jenkins-debian1) Wed, 13 Nov 2019 15:04:37 +0000 swh-loader-core (0.0.49-1~swh2) unstable-swh; urgency=medium * Update dependencies -- Antoine R. Dumont Fri, 08 Nov 2019 14:07:20 +0100 swh-loader-core (0.0.49-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.49 - (tagged by Antoine R. Dumont (@ardumont) on 2019-11-08 13:21:56 +0100) * Upstream changes: - v0.0.49 - New package loader implementations: archive, pypi, npm, deposit, debian -- Software Heritage autobuilder (on jenkins-debian1) Fri, 08 Nov 2019 12:29:47 +0000 swh-loader-core (0.0.48-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.48 - (tagged by Stefano Zacchiroli on 2019-10-01 16:49:39 +0200) * Upstream changes: - v0.0.48 - * typing: minimal changes to make a no-op mypy run pass -- Software Heritage autobuilder (on jenkins-debian1) Tue, 01 Oct 2019 14:52:59 +0000 swh-loader-core (0.0.47-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.47 - (tagged by Antoine Lambert on 2019-10-01 11:32:50 +0200) * Upstream changes: - version 0.0.47: Workaround HashCollision errors -- Software Heritage autobuilder (on jenkins-debian1) Tue, 01 Oct 2019 09:35:38 +0000 swh-loader-core (0.0.46-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.46 - (tagged by Antoine R. Dumont (@ardumont) on 2019-09-06 18:30:42 +0200) * Upstream changes: - v0.0.46 - pytest.ini: Remove warnings about our custom markers - pep8: Fix log.warning calls - core/loader: Fix get_save_data_path implementation - Fix validation errors in test. -- Software Heritage autobuilder (on jenkins-debian1) Fri, 06 Sep 2019 16:33:13 +0000 swh-loader-core (0.0.45-1~swh2) unstable-swh; urgency=medium * Fix missing build dependency -- Antoine R. Dumont (@ardumont) Tue, 03 Sep 2019 14:12:13 +0200 swh-loader-core (0.0.45-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.45 - (tagged by Antoine R. Dumont (@ardumont) on 2019-09-03 10:38:36 +0200) * Upstream changes: - v0.0.45 - loader: Provide visit type when calling origin_visit_add - loader: Drop keys 'perms' and 'path' from content before sending to the - storage - swh.loader.package: Implement GNU loader - docs: add code of conduct document -- Software Heritage autobuilder (on jenkins-debian1) Tue, 03 Sep 2019 08:41:49 +0000 swh-loader-core (0.0.44-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.44 - (tagged by Valentin Lorentz on 2019-06-25 12:18:27 +0200) * Upstream changes: - Drop use of deprecated methods fetch_history_* -- Software Heritage autobuilder (on jenkins-debian1) Wed, 26 Jun 2019 09:40:59 +0000 swh-loader-core (0.0.43-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.43 - (tagged by Valentin Lorentz on 2019-06-18 16:21:58 +0200) * Upstream changes: - Use origin urls instead of origin ids. -- Software Heritage autobuilder (on jenkins-debian1) Wed, 19 Jun 2019 09:33:53 +0000 swh-loader-core (0.0.42-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.42 - (tagged by David Douard on 2019-05-20 11:28:49 +0200) * Upstream changes: - v0.0.42 - update/fix requirements -- Software Heritage autobuilder (on jenkins-debian1) Mon, 20 May 2019 09:33:47 +0000 swh-loader-core (0.0.41-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.41 - (tagged by Antoine R. Dumont (@ardumont) on 2019-04-11 11:46:00 +0200) * Upstream changes: - v0.0.41 - core.loader: Migrate to latest snapshot_add, origin_visit_update api - core.loader: Count only the effectively new objects ingested - test_utils: Add coverage on utils module -- Software Heritage autobuilder (on jenkins-debian1) Thu, 11 Apr 2019 09:52:55 +0000 swh-loader-core (0.0.40-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.40 - (tagged by Antoine Lambert on 2019-03-29 10:57:14 +0100) * Upstream changes: - version 0.0.40 -- Software Heritage autobuilder (on jenkins-debian1) Fri, 29 Mar 2019 10:02:37 +0000 swh-loader-core (0.0.39-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.39 - (tagged by Antoine R. Dumont (@ardumont) on 2019-01-30 11:10:39 +0100) * Upstream changes: - v0.0.39 -- Software Heritage autobuilder (on jenkins-debian1) Wed, 30 Jan 2019 10:13:56 +0000 swh-loader-core (0.0.35-1~swh1) unstable-swh; urgency=medium * v0.0.35 * tests: Initialize tox.ini use * tests, debian/*: Migrate to pytest -- Antoine R. Dumont (@ardumont) Tue, 23 Oct 2018 15:47:22 +0200 swh-loader-core (0.0.34-1~swh1) unstable-swh; urgency=medium * v0.0.34 * setup: prepare for PyPI upload * README.md: Simplify module description * core.tests: Install tests fixture for derivative loaders to use -- Antoine R. Dumont (@ardumont) Tue, 09 Oct 2018 14:11:29 +0200 swh-loader-core (0.0.33-1~swh1) unstable-swh; urgency=medium * v0.0.33 * loader/utils: Add clean_dangling_folders function to ease clean up * loader/core: Add optional pre_cleanup for dangling files cleaning -- Antoine R. Dumont (@ardumont) Fri, 09 Mar 2018 14:41:17 +0100 swh-loader-core (0.0.32-1~swh1) unstable-swh; urgency=medium * v0.0.32 * Improve origin_visit initialization step * Properly sandbox the prepare statement so that if it breaks, we can * update appropriately the visit with the correct status -- Antoine R. Dumont (@ardumont) Wed, 07 Mar 2018 11:06:27 +0100 swh-loader-core (0.0.31-1~swh1) unstable-swh; urgency=medium * Release swh.loader.core v0.0.31 * Remove backwards-compatibility when sending snapshots -- Nicolas Dandrimont Tue, 13 Feb 2018 18:52:20 +0100 swh-loader-core (0.0.30-1~swh1) unstable-swh; urgency=medium * Release swh.loader.core v0.0.30 * Update Debian metadata for snapshot-related breakage -- Nicolas Dandrimont Tue, 06 Feb 2018 14:22:53 +0100 swh-loader-core (0.0.29-1~swh1) unstable-swh; urgency=medium * Release swh.loader.core v0.0.29 * Replace occurrences with snapshots * Enhance logging on error cases -- Nicolas Dandrimont Tue, 06 Feb 2018 14:13:11 +0100 swh-loader-core (0.0.28-1~swh1) unstable-swh; urgency=medium * v0.0.28 * Add stateless loader base class * Remove bare exception handlers -- Antoine R. Dumont (@ardumont) Tue, 19 Dec 2017 17:48:09 +0100 swh-loader-core (0.0.27-1~swh1) unstable-swh; urgency=medium * v0.0.27 * Migrate from indexer's indexer_configuration to storage's tool notion. -- Antoine R. Dumont (@ardumont) Thu, 07 Dec 2017 10:36:23 +0100 swh-loader-core (0.0.26-1~swh1) unstable-swh; urgency=medium * v0.0.26 * Fix send_provider method -- Antoine R. Dumont (@ardumont) Tue, 05 Dec 2017 15:40:57 +0100 swh-loader-core (0.0.25-1~swh1) unstable-swh; urgency=medium * v0.0.25 * swh.loader.core: Fix to retrieve the provider_id as an actual id * swh.loader.core: Fix log format error * swh.loader.core: Align log message according to conventions -- Antoine R. Dumont (@ardumont) Wed, 29 Nov 2017 12:55:45 +0100 swh-loader-core (0.0.24-1~swh1) unstable-swh; urgency=medium * v0.0.24 * Added metadata injection possible from loader core -- Antoine R. Dumont (@ardumont) Fri, 24 Nov 2017 11:35:40 +0100 swh-loader-core (0.0.23-1~swh1) unstable-swh; urgency=medium * v0.0.23 * loader: Fix dangling data flush -- Antoine R. Dumont (@ardumont) Tue, 07 Nov 2017 16:25:20 +0100 swh-loader-core (0.0.22-1~swh1) unstable-swh; urgency=medium * v0.0.22 * core.loader: Use the global setup set in swh.core.config * core.loader: Properly batch object insertions for big requests -- Antoine R. Dumont (@ardumont) Mon, 30 Oct 2017 18:50:00 +0100 swh-loader-core (0.0.21-1~swh1) unstable-swh; urgency=medium * v0.0.21 * swh.loader.core: Only send origin if not already sent before -- Antoine R. Dumont (@ardumont) Tue, 24 Oct 2017 16:30:53 +0200 swh-loader-core (0.0.20-1~swh1) unstable-swh; urgency=medium * v0.0.20 * Permit to add 'post_load' actions in loaders -- Antoine R. Dumont (@ardumont) Fri, 13 Oct 2017 14:30:37 +0200 swh-loader-core (0.0.19-1~swh1) unstable-swh; urgency=medium * v0.0.19 * Permit to add 'post_load' actions in loaders -- Antoine R. Dumont (@ardumont) Fri, 13 Oct 2017 14:14:14 +0200 swh-loader-core (0.0.18-1~swh1) unstable-swh; urgency=medium * Release swh.loader.core version 0.0.18 * Update packaging runes -- Nicolas Dandrimont Thu, 12 Oct 2017 18:07:53 +0200 swh-loader-core (0.0.17-1~swh1) unstable-swh; urgency=medium * Release swh.loader.core v0.0.17 * Allow iterating when fetching and storing data * Allow overriding the status of the loaded visit * Allow overriding the status of the load itself -- Nicolas Dandrimont Wed, 11 Oct 2017 16:38:29 +0200 swh-loader-core (0.0.16-1~swh1) unstable-swh; urgency=medium * Release swh.loader.core v0.0.16 * Migrate from swh.model.git to swh.model.from_disk -- Nicolas Dandrimont Fri, 06 Oct 2017 14:46:41 +0200 swh-loader-core (0.0.15-1~swh1) unstable-swh; urgency=medium * v0.0.15 * docs: Add sphinx apidoc generation skeleton * docs: Add a simple README.md explaining the module's goal * swh.loader.core.loader: Unify origin_visit add/update function call -- Antoine R. Dumont (@ardumont) Fri, 29 Sep 2017 11:47:37 +0200 swh-loader-core (0.0.14-1~swh1) unstable-swh; urgency=medium * v0.0.14 * Add the blake2s256 hash computation -- Antoine R. Dumont (@ardumont) Sat, 25 Mar 2017 18:20:52 +0100 swh-loader-core (0.0.13-1~swh1) unstable-swh; urgency=medium * v0.0.13 * Improve core loader's interface api -- Antoine R. Dumont (@ardumont) Wed, 22 Feb 2017 13:43:54 +0100 swh-loader-core (0.0.12-1~swh1) unstable-swh; urgency=medium * v0.0.12 * Update storage configuration reading -- Antoine R. Dumont (@ardumont) Thu, 15 Dec 2016 18:34:41 +0100 swh-loader-core (0.0.11-1~swh1) unstable-swh; urgency=medium * v0.0.11 * d/control: Bump dependency to latest storage * Fix: Objects can be injected even though global loading failed * Populate the counters in fetch_history * Open open/close fetch_history function in the core loader -- Antoine R. Dumont (@ardumont) Wed, 24 Aug 2016 14:38:55 +0200 swh-loader-core (0.0.10-1~swh1) unstable-swh; urgency=medium * v0.0.10 * d/control: Update dependency -- Antoine R. Dumont (@ardumont) Sat, 11 Jun 2016 02:26:50 +0200 swh-loader-core (0.0.9-1~swh1) unstable-swh; urgency=medium * v0.0.9 * Improve default task that initialize storage as well -- Antoine R. Dumont (@ardumont) Fri, 10 Jun 2016 15:12:14 +0200 swh-loader-core (0.0.8-1~swh1) unstable-swh; urgency=medium * v0.0.8 * Migrate specific converter to the right module * Fix dangling parameter -- Antoine R. Dumont (@ardumont) Wed, 08 Jun 2016 18:09:23 +0200 swh-loader-core (0.0.7-1~swh1) unstable-swh; urgency=medium * v0.0.7 * Fix on revision conversion -- Antoine R. Dumont (@ardumont) Wed, 08 Jun 2016 16:19:02 +0200 swh-loader-core (0.0.6-1~swh1) unstable-swh; urgency=medium * v0.0.6 * d/control: Bump dependency on swh-model * d/control: Add missing description * Keep the abstraction for all entities * Align parameter definition order * Fix missing option in DEFAULT ones * Decrease verbosity * Fix missing origin_id assignment * d/rules: Add target to run tests during packaging -- Antoine R. Dumont (@ardumont) Wed, 08 Jun 2016 16:00:40 +0200 swh-loader-core (0.0.5-1~swh1) unstable-swh; urgency=medium * v0.0.5 -- Antoine R. Dumont (@ardumont) Wed, 25 May 2016 12:17:06 +0200 swh-loader-core (0.0.4-1~swh1) unstable-swh; urgency=medium * v0.0.4 * Rename package from python3-swh.loader to python3-swh.loader.core -- Antoine R. Dumont (@ardumont) Wed, 25 May 2016 11:44:48 +0200 swh-loader-core (0.0.3-1~swh1) unstable-swh; urgency=medium * v0.0.3 * Improve default configuration * Rename package from swh-loader-vcs to swh-loader -- Antoine R. Dumont (@ardumont) Wed, 25 May 2016 11:23:06 +0200 swh-loader-core (0.0.2-1~swh1) unstable-swh; urgency=medium * v0.0.2 * Fix: Flush data even when no data is sent to swh-storage -- Antoine R. Dumont (@ardumont) Tue, 24 May 2016 16:41:49 +0200 swh-loader-core (0.0.1-1~swh1) unstable-swh; urgency=medium * Initial release * v0.0.1 -- Antoine R. Dumont (@ardumont) Wed, 13 Apr 2016 16:54:47 +0200 diff --git a/requirements.txt b/requirements.txt index 644a170..87e7f99 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,12 +1,13 @@ # Add here external Python modules dependencies, one per line. Module names # should match https://pypi.python.org/pypi names. For the full spec or # dependency lines, see https://pip.readthedocs.org/en/1.1/requirements.html psutil requests iso8601 pkginfo python-debian python-dateutil typing-extensions toml +packaging diff --git a/swh.loader.core.egg-info/PKG-INFO b/swh.loader.core.egg-info/PKG-INFO index 0de803a..1462b4d 100644 --- a/swh.loader.core.egg-info/PKG-INFO +++ b/swh.loader.core.egg-info/PKG-INFO @@ -1,52 +1,52 @@ Metadata-Version: 2.1 Name: swh.loader.core -Version: 4.0.0 +Version: 4.1.0 Summary: Software Heritage Base Loader Home-page: https://forge.softwareheritage.org/diffusion/DLDBASE Author: Software Heritage developers Author-email: swh-devel@inria.fr Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest Project-URL: Funding, https://www.softwareheritage.org/donate Project-URL: Source, https://forge.softwareheritage.org/source/swh-loader-core Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-loader-core/ Classifier: Programming Language :: Python :: 3 Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Classifier: Operating System :: OS Independent Classifier: Development Status :: 5 - Production/Stable Requires-Python: >=3.7 Description-Content-Type: text/markdown Provides-Extra: testing License-File: LICENSE License-File: AUTHORS Software Heritage - Loader foundations ====================================== The Software Heritage Loader Core is a low-level loading utilities and helpers used by :term:`loaders `. The main entry points are classes: - :class:`swh.loader.core.loader.BaseLoader` for loaders (e.g. svn) - :class:`swh.loader.core.loader.DVCSLoader` for DVCS loaders (e.g. hg, git, ...) - :class:`swh.loader.package.loader.PackageLoader` for Package loaders (e.g. PyPI, Npm, ...) Package loaders --------------- This package also implements many package loaders directly, out of convenience, as they usually are quite similar and each fits in a single file. They all roughly follow these steps, explained in the :py:meth:`swh.loader.package.loader.PackageLoader.load` documentation. See the :ref:`package-loader-tutorial` for details. VCS loaders ----------- Unlike package loaders, VCS loaders remain in separate packages, as they often need more advanced conversions and very VCS-specific operations. This usually involves getting the branches of a repository and recursively loading revisions in the history (and directory trees in these revisions), until a known revision is found diff --git a/swh.loader.core.egg-info/SOURCES.txt b/swh.loader.core.egg-info/SOURCES.txt index 0bbd56c..eac17d5 100644 --- a/swh.loader.core.egg-info/SOURCES.txt +++ b/swh.loader.core.egg-info/SOURCES.txt @@ -1,287 +1,288 @@ .git-blame-ignore-revs .gitignore .pre-commit-config.yaml AUTHORS CODE_OF_CONDUCT.md CONTRIBUTORS LICENSE MANIFEST.in Makefile README.rst conftest.py mypy.ini pyproject.toml pytest.ini requirements-swh.txt requirements-test.txt requirements.txt setup.cfg setup.py tox.ini docs/.gitignore docs/Makefile docs/README.rst docs/cli.rst docs/conf.py docs/index.rst docs/package-loader-specifications.rst docs/package-loader-tutorial.rst docs/vcs-loader-overview.rst docs/_static/.placeholder docs/_templates/.placeholder swh/__init__.py swh.loader.core.egg-info/PKG-INFO swh.loader.core.egg-info/SOURCES.txt swh.loader.core.egg-info/dependency_links.txt swh.loader.core.egg-info/entry_points.txt swh.loader.core.egg-info/requires.txt swh.loader.core.egg-info/top_level.txt swh/loader/__init__.py swh/loader/cli.py swh/loader/exception.py swh/loader/pytest_plugin.py swh/loader/core/__init__.py swh/loader/core/converters.py swh/loader/core/loader.py swh/loader/core/metadata_fetchers.py swh/loader/core/py.typed swh/loader/core/utils.py swh/loader/core/tests/__init__.py swh/loader/core/tests/test_converters.py swh/loader/core/tests/test_loader.py swh/loader/core/tests/test_utils.py swh/loader/package/__init__.py swh/loader/package/loader.py swh/loader/package/py.typed swh/loader/package/utils.py swh/loader/package/arch/__init__.py swh/loader/package/arch/loader.py swh/loader/package/arch/tasks.py swh/loader/package/arch/tests/__init__.py swh/loader/package/arch/tests/test_arch.py swh/loader/package/arch/tests/test_tasks.py swh/loader/package/arch/tests/data/fake_arch.sh swh/loader/package/arch/tests/data/https_archive.archlinux.org/packages_d_dialog_dialog-1:1.3_20190211-1-x86_64.pkg.tar.xz swh/loader/package/arch/tests/data/https_archive.archlinux.org/packages_d_dialog_dialog-1:1.3_20220414-1-x86_64.pkg.tar.zst swh/loader/package/arch/tests/data/https_uk.mirror.archlinuxarm.org/aarch64_core_gzip-1.12-1-aarch64.pkg.tar.xz swh/loader/package/archive/__init__.py swh/loader/package/archive/loader.py swh/loader/package/archive/tasks.py swh/loader/package/archive/tests/__init__.py swh/loader/package/archive/tests/test_archive.py swh/loader/package/archive/tests/test_tasks.py swh/loader/package/archive/tests/data/not_gzipped_tarball.tar.gz swh/loader/package/archive/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.1.0.tar.gz swh/loader/package/archive/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.1.0.tar.gz_visit1 swh/loader/package/archive/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.1.0.tar.gz_visit2 swh/loader/package/archive/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.2.0.tar.gz swh/loader/package/aur/__init__.py swh/loader/package/aur/loader.py swh/loader/package/aur/tasks.py swh/loader/package/aur/tests/__init__.py swh/loader/package/aur/tests/test_aur.py swh/loader/package/aur/tests/test_tasks.py swh/loader/package/aur/tests/data/fake_aur.sh swh/loader/package/aur/tests/data/https_aur.archlinux.org/cgit_aur.git_snapshot_a-fake-one.tar.gz swh/loader/package/aur/tests/data/https_aur.archlinux.org/cgit_aur.git_snapshot_hg-evolve.tar.gz swh/loader/package/aur/tests/data/https_aur.archlinux.org/cgit_aur.git_snapshot_ibus-git.tar.gz swh/loader/package/aur/tests/data/https_aur.archlinux.org/cgit_aur.git_snapshot_libervia-web-hg.tar.gz swh/loader/package/aur/tests/data/https_aur.archlinux.org/cgit_aur.git_snapshot_tealdeer-git.tar.gz swh/loader/package/cran/__init__.py swh/loader/package/cran/loader.py swh/loader/package/cran/tasks.py swh/loader/package/cran/tests/__init__.py swh/loader/package/cran/tests/test_cran.py swh/loader/package/cran/tests/test_tasks.py swh/loader/package/cran/tests/data/description/KnownBR swh/loader/package/cran/tests/data/description/acepack swh/loader/package/cran/tests/data/https_cran.r-project.org/src_contrib_1.4.0_Recommended_KernSmooth_2.22-6.tar.gz swh/loader/package/crates/__init__.py swh/loader/package/crates/loader.py swh/loader/package/crates/tasks.py swh/loader/package/crates/tests/__init__.py swh/loader/package/crates/tests/test_crates.py swh/loader/package/crates/tests/test_tasks.py swh/loader/package/crates/tests/data/fake_crates.sh swh/loader/package/crates/tests/data/https_crates.io/api_v1_crates_hg-core swh/loader/package/crates/tests/data/https_crates.io/api_v1_crates_micro-timer swh/loader/package/crates/tests/data/https_static.crates.io/crates_hg-core_hg-core-0.0.1.crate swh/loader/package/crates/tests/data/https_static.crates.io/crates_micro-timer_micro-timer-0.1.0.crate swh/loader/package/crates/tests/data/https_static.crates.io/crates_micro-timer_micro-timer-0.1.1.crate swh/loader/package/crates/tests/data/https_static.crates.io/crates_micro-timer_micro-timer-0.1.2.crate swh/loader/package/crates/tests/data/https_static.crates.io/crates_micro-timer_micro-timer-0.2.0.crate swh/loader/package/crates/tests/data/https_static.crates.io/crates_micro-timer_micro-timer-0.2.1.crate swh/loader/package/crates/tests/data/https_static.crates.io/crates_micro-timer_micro-timer-0.3.0.crate swh/loader/package/crates/tests/data/https_static.crates.io/crates_micro-timer_micro-timer-0.3.1.crate swh/loader/package/crates/tests/data/https_static.crates.io/crates_micro-timer_micro-timer-0.4.0.crate swh/loader/package/debian/__init__.py swh/loader/package/debian/loader.py swh/loader/package/debian/tasks.py swh/loader/package/debian/tests/__init__.py swh/loader/package/debian/tests/test_debian.py swh/loader/package/debian/tests/test_tasks.py swh/loader/package/debian/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-3.diff.gz swh/loader/package/debian/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-3.dsc swh/loader/package/debian/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-4.diff.gz swh/loader/package/debian/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2-4.dsc swh/loader/package/debian/tests/data/http_deb.debian.org/debian_pool_contrib_c_cicero_cicero_0.7.2.orig.tar.gz swh/loader/package/debian/tests/data/http_deb.debian.org/onefile.txt swh/loader/package/deposit/__init__.py swh/loader/package/deposit/loader.py swh/loader/package/deposit/tasks.py swh/loader/package/deposit/tests/__init__.py swh/loader/package/deposit/tests/conftest.py swh/loader/package/deposit/tests/test_deposit.py swh/loader/package/deposit/tests/test_tasks.py swh/loader/package/deposit/tests/data/https_deposit.softwareheritage.org/1_private_666_meta swh/loader/package/deposit/tests/data/https_deposit.softwareheritage.org/1_private_666_raw swh/loader/package/deposit/tests/data/https_deposit.softwareheritage.org/1_private_777_meta swh/loader/package/deposit/tests/data/https_deposit.softwareheritage.org/1_private_777_raw swh/loader/package/deposit/tests/data/https_deposit.softwareheritage.org/1_private_888_meta swh/loader/package/deposit/tests/data/https_deposit.softwareheritage.org/1_private_888_raw swh/loader/package/deposit/tests/data/https_deposit.softwareheritage.org/1_private_999_meta swh/loader/package/deposit/tests/data/https_deposit.softwareheritage.org/1_private_999_raw swh/loader/package/deposit/tests/data/https_deposit.softwareheritage.org/hello-2.10.zip swh/loader/package/deposit/tests/data/https_deposit.softwareheritage.org/hello-2.12.tar.gz swh/loader/package/deposit/tests/data/https_deposit.softwareheritage.org/hello_2.10.json swh/loader/package/deposit/tests/data/https_deposit.softwareheritage.org/hello_2.11.json swh/loader/package/deposit/tests/data/https_deposit.softwareheritage.org/hello_2.12.json swh/loader/package/deposit/tests/data/https_deposit.softwareheritage.org/hello_2.13.json swh/loader/package/golang/__init__.py swh/loader/package/golang/loader.py swh/loader/package/golang/tasks.py swh/loader/package/golang/tests/__init__.py swh/loader/package/golang/tests/test_golang.py swh/loader/package/golang/tests/test_tasks.py swh/loader/package/golang/tests/data/https_proxy.golang.org/example.com_basic-go-module_@latest swh/loader/package/golang/tests/data/https_proxy.golang.org/example.com_basic-go-module_@v_list swh/loader/package/golang/tests/data/https_proxy.golang.org/example.com_basic-go-module_@v_v0.1.3.info swh/loader/package/golang/tests/data/https_proxy.golang.org/example.com_basic-go-module_@v_v0.1.3.zip swh/loader/package/maven/__init__.py swh/loader/package/maven/loader.py swh/loader/package/maven/tasks.py swh/loader/package/maven/tests/__init__.py swh/loader/package/maven/tests/test_maven.py swh/loader/package/maven/tests/test_tasks.py swh/loader/package/maven/tests/data/https_maven.org/sprova4j-0.1.0-sources.jar swh/loader/package/maven/tests/data/https_maven.org/sprova4j-0.1.0.pom swh/loader/package/maven/tests/data/https_maven.org/sprova4j-0.1.1-sources.jar swh/loader/package/maven/tests/data/https_maven.org/sprova4j-0.1.1.pom swh/loader/package/nixguix/__init__.py swh/loader/package/nixguix/loader.py swh/loader/package/nixguix/tasks.py swh/loader/package/nixguix/tests/__init__.py swh/loader/package/nixguix/tests/conftest.py swh/loader/package/nixguix/tests/test_nixguix.py swh/loader/package/nixguix/tests/test_tasks.py swh/loader/package/nixguix/tests/data/https_example.com/file.txt swh/loader/package/nixguix/tests/data/https_fail.com/truncated-archive.tgz swh/loader/package/nixguix/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.1.0.tar.gz swh/loader/package/nixguix/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.1.0.tar.gz_visit1 swh/loader/package/nixguix/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.1.0.tar.gz_visit2 swh/loader/package/nixguix/tests/data/https_ftp.gnu.org/gnu_8sync_8sync-0.2.0.tar.gz swh/loader/package/nixguix/tests/data/https_github.com/owner-1_repository-1_revision-1.tgz swh/loader/package/nixguix/tests/data/https_github.com/owner-2_repository-1_revision-1.tgz swh/loader/package/nixguix/tests/data/https_github.com/owner-3_repository-1_revision-1.tgz swh/loader/package/nixguix/tests/data/https_nix-community.github.io/nixpkgs-swh_sources-EOFError.json swh/loader/package/nixguix/tests/data/https_nix-community.github.io/nixpkgs-swh_sources.json swh/loader/package/nixguix/tests/data/https_nix-community.github.io/nixpkgs-swh_sources.json_visit1 swh/loader/package/nixguix/tests/data/https_nix-community.github.io/nixpkgs-swh_sources_special.json swh/loader/package/nixguix/tests/data/https_nix-community.github.io/nixpkgs-swh_sources_special.json_visit1 swh/loader/package/npm/__init__.py swh/loader/package/npm/loader.py swh/loader/package/npm/tasks.py swh/loader/package/npm/tests/__init__.py swh/loader/package/npm/tests/test_npm.py swh/loader/package/npm/tests/test_tasks.py swh/loader/package/npm/tests/data/https_registry.npmjs.org/@aller_shared_-_shared-0.1.0.tgz swh/loader/package/npm/tests/data/https_registry.npmjs.org/@aller_shared_-_shared-0.1.1-alpha.14.tgz swh/loader/package/npm/tests/data/https_registry.npmjs.org/jammit-express_-_jammit-express-0.0.1.tgz swh/loader/package/npm/tests/data/https_registry.npmjs.org/nativescript-telerik-analytics_-_nativescript-telerik-analytics-1.0.0.tgz swh/loader/package/npm/tests/data/https_registry.npmjs.org/org_-_org-0.0.2.tgz swh/loader/package/npm/tests/data/https_registry.npmjs.org/org_-_org-0.0.3-beta.tgz swh/loader/package/npm/tests/data/https_registry.npmjs.org/org_-_org-0.0.3.tgz swh/loader/package/npm/tests/data/https_registry.npmjs.org/org_-_org-0.0.4.tgz swh/loader/package/npm/tests/data/https_registry.npmjs.org/org_-_org-0.0.5.tgz swh/loader/package/npm/tests/data/https_registry.npmjs.org/org_-_org-0.1.0.tgz swh/loader/package/npm/tests/data/https_registry.npmjs.org/org_-_org-0.2.0.tgz swh/loader/package/npm/tests/data/https_replicate.npmjs.com/@aller_shared swh/loader/package/npm/tests/data/https_replicate.npmjs.com/catify swh/loader/package/npm/tests/data/https_replicate.npmjs.com/jammit-express swh/loader/package/npm/tests/data/https_replicate.npmjs.com/jammit-no-time swh/loader/package/npm/tests/data/https_replicate.npmjs.com/nativescript-telerik-analytics swh/loader/package/npm/tests/data/https_replicate.npmjs.com/org swh/loader/package/npm/tests/data/https_replicate.npmjs.com/org_version_mismatch swh/loader/package/npm/tests/data/https_replicate.npmjs.com/org_visit1 swh/loader/package/opam/__init__.py swh/loader/package/opam/loader.py swh/loader/package/opam/tasks.py swh/loader/package/opam/tests/__init__.py swh/loader/package/opam/tests/test_opam.py swh/loader/package/opam/tests/test_tasks.py swh/loader/package/opam/tests/data/fake_opam_repo/_repo swh/loader/package/opam/tests/data/fake_opam_repo/version swh/loader/package/opam/tests/data/fake_opam_repo/repo/loadertest/lock swh/loader/package/opam/tests/data/fake_opam_repo/repo/loadertest/repos-config swh/loader/package/opam/tests/data/fake_opam_repo/repo/loadertest/packages/agrid/agrid.0.1/opam swh/loader/package/opam/tests/data/fake_opam_repo/repo/loadertest/packages/directories/directories.0.1/opam swh/loader/package/opam/tests/data/fake_opam_repo/repo/loadertest/packages/directories/directories.0.2/opam swh/loader/package/opam/tests/data/fake_opam_repo/repo/loadertest/packages/directories/directories.0.3/opam swh/loader/package/opam/tests/data/fake_opam_repo/repo/loadertest/packages/ocb/ocb.0.1/opam swh/loader/package/opam/tests/data/https_github.com/OCamlPro_agrid_archive_0.1.tar.gz swh/loader/package/opam/tests/data/https_github.com/OCamlPro_directories_archive_0.1.tar.gz swh/loader/package/opam/tests/data/https_github.com/OCamlPro_directories_archive_0.2.tar.gz swh/loader/package/opam/tests/data/https_github.com/OCamlPro_directories_archive_0.3.tar.gz swh/loader/package/opam/tests/data/https_github.com/OCamlPro_ocb_archive_0.1.tar.gz swh/loader/package/pubdev/__init__.py swh/loader/package/pubdev/loader.py swh/loader/package/pubdev/tasks.py swh/loader/package/pubdev/tests/__init__.py swh/loader/package/pubdev/tests/test_pubdev.py swh/loader/package/pubdev/tests/test_tasks.py swh/loader/package/pubdev/tests/data/fake_pubdev.sh swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_Autolinker_versions_0.1.1.tar.gz swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_authentication_versions_0.0.1.tar.gz swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_bezier_versions_1.1.5.tar.gz swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_pdf_versions_1.0.0.tar.gz swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_pdf_versions_3.8.2.tar.gz swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_Autolinker swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_abstract_io +swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_audio_manager swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_authentication swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_bezier swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_pdf swh/loader/package/pypi/__init__.py swh/loader/package/pypi/loader.py swh/loader/package/pypi/tasks.py swh/loader/package/pypi/tests/__init__.py swh/loader/package/pypi/tests/test_pypi.py swh/loader/package/pypi/tests/test_tasks.py swh/loader/package/pypi/tests/data/https_files.pythonhosted.org/0805nexter-1.1.0.tar.gz swh/loader/package/pypi/tests/data/https_files.pythonhosted.org/0805nexter-1.1.0.zip swh/loader/package/pypi/tests/data/https_files.pythonhosted.org/0805nexter-1.2.0.zip swh/loader/package/pypi/tests/data/https_files.pythonhosted.org/0805nexter-1.3.0.zip swh/loader/package/pypi/tests/data/https_files.pythonhosted.org/0805nexter-1.4.0.zip swh/loader/package/pypi/tests/data/https_files.pythonhosted.org/nexter-1.1.0.tar.gz swh/loader/package/pypi/tests/data/https_files.pythonhosted.org/nexter-1.1.0.zip swh/loader/package/pypi/tests/data/https_files.pythonhosted.org/packages_70_97_c49fb8ec24a7aaab54c3dbfbb5a6ca1431419d9ee0f6c363d9ad01d2b8b1_0805nexter-1.3.0.zip swh/loader/package/pypi/tests/data/https_files.pythonhosted.org/packages_86_10_c9555ec63106153aaaad753a281ff47f4ac79e980ff7f5d740d6649cd56a_upymenu-0.0.1.tar.gz swh/loader/package/pypi/tests/data/https_files.pythonhosted.org/packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip swh/loader/package/pypi/tests/data/https_files.pythonhosted.org/packages_c4_a0_4562cda161dc4ecbbe9e2a11eb365400c0461845c5be70d73869786809c4_0805nexter-1.2.0.zip_visit1 swh/loader/package/pypi/tests/data/https_files.pythonhosted.org/packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip swh/loader/package/pypi/tests/data/https_files.pythonhosted.org/packages_ec_65_c0116953c9a3f47de89e71964d6c7b0c783b01f29fa3390584dbf3046b4d_0805nexter-1.1.0.zip_visit1 swh/loader/package/pypi/tests/data/https_pypi.org/pypi_0805nexter_json swh/loader/package/pypi/tests/data/https_pypi.org/pypi_0805nexter_json_visit1 swh/loader/package/pypi/tests/data/https_pypi.org/pypi_nexter_json swh/loader/package/pypi/tests/data/https_pypi.org/pypi_upymenu_json swh/loader/package/tests/__init__.py swh/loader/package/tests/common.py swh/loader/package/tests/test_conftest.py swh/loader/package/tests/test_loader.py swh/loader/package/tests/test_loader_metadata.py swh/loader/package/tests/test_utils.py swh/loader/tests/__init__.py swh/loader/tests/conftest.py swh/loader/tests/py.typed swh/loader/tests/test_cli.py swh/loader/tests/test_init.py swh/loader/tests/data/0805nexter-1.1.0.tar.gz \ No newline at end of file diff --git a/swh.loader.core.egg-info/requires.txt b/swh.loader.core.egg-info/requires.txt index fe872fe..bfad1d7 100644 --- a/swh.loader.core.egg-info/requires.txt +++ b/swh.loader.core.egg-info/requires.txt @@ -1,25 +1,26 @@ psutil requests iso8601 pkginfo python-debian python-dateutil typing-extensions toml +packaging swh.core>=2.12 swh.model>=4.4.0 swh.objstorage>=0.2.2 swh.scheduler>=0.4.0 swh.storage>=0.29.0 [testing] pytest pytest-mock requests_mock swh-core[testing] swh-scheduler[testing]>=0.5.0 swh-storage[testing]>=0.10.6 types-click types-python-dateutil types-pyyaml types-requests diff --git a/swh/loader/core/loader.py b/swh/loader/core/loader.py index 30e30f6..c787d2e 100644 --- a/swh/loader/core/loader.py +++ b/swh/loader/core/loader.py @@ -1,636 +1,642 @@ # Copyright (C) 2015-2022 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU General Public License version 3, or any later version # See top-level LICENSE file for more information import datetime import hashlib import logging import os import time from typing import Any, ContextManager, Dict, Iterable, List, Optional, Union import sentry_sdk from swh.core.config import load_from_envvar from swh.core.statsd import Statsd from swh.loader.core.metadata_fetchers import CredentialsType, get_fetchers_for_lister from swh.loader.exception import NotFound from swh.model.model import ( BaseContent, Content, Directory, Origin, OriginVisit, OriginVisitStatus, RawExtrinsicMetadata, Release, Revision, Sha1Git, SkippedContent, Snapshot, ) from swh.storage import get_storage from swh.storage.interface import StorageInterface from swh.storage.utils import now DEFAULT_CONFIG: Dict[str, Any] = { "max_content_size": 100 * 1024 * 1024, } +SENTRY_ORIGIN_URL_TAG_NAME = "swh.loader.origin_url" +SENTRY_VISIT_TYPE_TAG_NAME = "swh.loader.visit_type" + class BaseLoader: """Base class for (D)VCS loaders (e.g Svn, Git, Mercurial, ...) or PackageLoader (e.g PyPI, Npm, CRAN, ...) A loader retrieves origin information (git/mercurial/svn repositories, pypi/npm/... package artifacts), ingests the contents/directories/revisions/releases/snapshot read from those artifacts and send them to the archive through the storage backend. The main entry point for the loader is the :func:`load` function. 2 static methods (:func:`from_config`, :func:`from_configfile`) centralizes and eases the loader instantiation from either configuration dict or configuration file. Some class examples: - :class:`SvnLoader` - :class:`GitLoader` - :class:`PyPILoader` - :class:`NpmLoader` Args: lister_name: Name of the lister which triggered this load. If provided, the loader will try to use the forge's API to retrieve extrinsic metadata lister_instance_name: Name of the lister instance which triggered this load. Must be None iff lister_name is, but it may be the empty string for listers with a single instance. """ visit_type: str origin: Origin loaded_snapshot_id: Optional[Sha1Git] parent_origins: Optional[List[Origin]] """If the given origin is a "forge fork" (ie. created with the "Fork" button of GitHub-like forges), :meth:`build_extrinsic_origin_metadata` sets this to a list of origins it was forked from; closest parent first.""" def __init__( self, storage: StorageInterface, origin_url: str, logging_class: Optional[str] = None, save_data_path: Optional[str] = None, max_content_size: Optional[int] = None, lister_name: Optional[str] = None, lister_instance_name: Optional[str] = None, metadata_fetcher_credentials: CredentialsType = None, ): if lister_name == "": raise ValueError("lister_name must not be the empty string") if lister_name is None and lister_instance_name is not None: raise ValueError( f"lister_name is None but lister_instance_name is {lister_instance_name!r}" ) if lister_name is not None and lister_instance_name is None: raise ValueError( f"lister_instance_name is None but lister_name is {lister_name!r}" ) self.storage = storage self.origin = Origin(url=origin_url) self.max_content_size = int(max_content_size) if max_content_size else None self.lister_name = lister_name self.lister_instance_name = lister_instance_name self.metadata_fetcher_credentials = metadata_fetcher_credentials or {} if logging_class is None: logging_class = "%s.%s" % ( self.__class__.__module__, self.__class__.__name__, ) self.log = logging.getLogger(logging_class) _log = logging.getLogger("requests.packages.urllib3.connectionpool") _log.setLevel(logging.WARN) + sentry_sdk.set_tag(SENTRY_ORIGIN_URL_TAG_NAME, self.origin.url) + sentry_sdk.set_tag(SENTRY_VISIT_TYPE_TAG_NAME, self.visit_type) + # possibly overridden in self.prepare method self.visit_date = datetime.datetime.now(tz=datetime.timezone.utc) self.loaded_snapshot_id = None if save_data_path: path = save_data_path os.stat(path) if not os.access(path, os.R_OK | os.W_OK): raise PermissionError("Permission denied: %r" % path) self.save_data_path = save_data_path self.parent_origins = None self.statsd = Statsd( namespace="swh_loader", constant_tags={"visit_type": self.visit_type} ) @classmethod def from_config(cls, storage: Dict[str, Any], **config: Any): """Instantiate a loader from a configuration dict. This is basically a backwards-compatibility shim for the CLI. Args: storage: instantiation config for the storage config: the configuration dict for the loader, with the following keys: - credentials (optional): credentials list for the scheduler - any other kwargs passed to the loader. Returns: the instantiated loader """ # Drop the legacy config keys which aren't used for this generation of loader. for legacy_key in ("storage", "celery"): config.pop(legacy_key, None) # Instantiate the storage storage_instance = get_storage(**storage) return cls(storage=storage_instance, **config) @classmethod def from_configfile(cls, **kwargs: Any): """Instantiate a loader from the configuration loaded from the SWH_CONFIG_FILENAME envvar, with potential extra keyword arguments if their value is not None. Args: kwargs: kwargs passed to the loader instantiation """ config = dict(load_from_envvar(DEFAULT_CONFIG)) config.update({k: v for k, v in kwargs.items() if v is not None}) return cls.from_config(**config) def save_data(self) -> None: """Save the data associated to the current load""" raise NotImplementedError def get_save_data_path(self) -> str: """The path to which we archive the loader's raw data""" if not hasattr(self, "__save_data_path"): year = str(self.visit_date.year) assert self.origin url = self.origin.url.encode("utf-8") origin_url_hash = hashlib.sha1(url).hexdigest() path = "%s/sha1:%s/%s/%s" % ( self.save_data_path, origin_url_hash[0:2], origin_url_hash, year, ) os.makedirs(path, exist_ok=True) self.__save_data_path = path return self.__save_data_path def flush(self) -> Dict[str, int]: """Flush any potential buffered data not sent to swh-storage. Returns the same value as :meth:`swh.storage.interface.StorageInterface.flush`. """ return self.storage.flush() def cleanup(self) -> None: """Last step executed by the loader.""" raise NotImplementedError def _store_origin_visit(self) -> None: """Store origin and visit references. Sets the self.visit references.""" assert self.origin self.storage.origin_add([self.origin]) assert isinstance(self.visit_type, str) self.visit = list( self.storage.origin_visit_add( [ OriginVisit( origin=self.origin.url, date=self.visit_date, type=self.visit_type, ) ] ) )[0] def prepare(self) -> None: """Second step executed by the loader to prepare some state needed by the loader. Raises NotFound exception if the origin to ingest is not found. """ raise NotImplementedError def get_origin(self) -> Origin: """Get the origin that is currently being loaded. self.origin should be set in :func:`prepare_origin` Returns: dict: an origin ready to be sent to storage by :func:`origin_add`. """ assert self.origin return self.origin def fetch_data(self) -> bool: """Fetch the data from the source the loader is currently loading (ex: git/hg/svn/... repository). Returns: a value that is interpreted as a boolean. If True, fetch_data needs to be called again to complete loading. """ raise NotImplementedError def process_data(self) -> bool: """Run any additional processing between fetching and storing the data Returns: a value that is interpreted as a boolean. If True, fetch_data needs to be called again to complete loading. Ignored if ``fetch_data`` already returned :const:`False`. """ return True def store_data(self): """Store fetched data in the database. Should call the :func:`maybe_load_xyz` methods, which handle the bundles sent to storage, rather than send directly. """ raise NotImplementedError def load_status(self) -> Dict[str, str]: """Detailed loading status. Defaults to logging an eventful load. Returns: a dictionary that is eventually passed back as the task's result to the scheduler, allowing tuning of the task recurrence mechanism. """ return { "status": "eventful", } def post_load(self, success: bool = True) -> None: """Permit the loader to do some additional actions according to status after the loading is done. The flag success indicates the loading's status. Defaults to doing nothing. This is up to the implementer of this method to make sure this does not break. Args: success (bool): the success status of the loading """ pass def visit_status(self) -> str: """Detailed visit status. Defaults to logging a full visit. """ return "full" def pre_cleanup(self) -> None: """As a first step, will try and check for dangling data to cleanup. This should do its best to avoid raising issues. """ pass def load(self) -> Dict[str, str]: r"""Loading logic for the loader to follow: - Store the actual ``origin_visit`` to storage - Call :meth:`prepare` to prepare any eventual state - Call :meth:`get_origin` to get the origin we work with and store - while True: - Call :meth:`fetch_data` to fetch the data to store - Call :meth:`process_data` to optionally run processing between :meth:`fetch_data` and :meth:`store_data` - Call :meth:`store_data` to store the data - Call :meth:`cleanup` to clean up any eventual state put in place in :meth:`prepare` method. """ try: with self.statsd_timed("pre_cleanup"): self.pre_cleanup() except Exception: msg = "Cleaning up dangling data failed! Continue loading." self.log.warning(msg) sentry_sdk.capture_exception() self._store_origin_visit() assert ( self.visit.visit ), "The method `_store_origin_visit` should set the visit (OriginVisit)" self.log.info( "Load origin '%s' with type '%s'", self.origin.url, self.visit.type ) try: with self.statsd_timed("build_extrinsic_origin_metadata"): metadata = self.build_extrinsic_origin_metadata() self.load_metadata_objects(metadata) except Exception as e: sentry_sdk.capture_exception(e) # Do not fail the whole task if this is the only failure self.log.exception( "Failure while loading extrinsic origin metadata.", extra={ "swh_task_args": [], "swh_task_kwargs": { "origin": self.origin.url, "lister_name": self.lister_name, "lister_instance_name": self.lister_instance_name, }, }, ) total_time_fetch_data = 0.0 total_time_process_data = 0.0 total_time_store_data = 0.0 # Initially not a success, will be True when actually one status = "failed" success = False try: with self.statsd_timed("prepare"): self.prepare() while True: t1 = time.monotonic() more_data_to_fetch = self.fetch_data() t2 = time.monotonic() total_time_fetch_data += t2 - t1 more_data_to_fetch = self.process_data() and more_data_to_fetch t3 = time.monotonic() total_time_process_data += t3 - t2 self.store_data() t4 = time.monotonic() total_time_store_data += t4 - t3 if not more_data_to_fetch: break self.statsd_timing("fetch_data", total_time_fetch_data * 1000.0) self.statsd_timing("process_data", total_time_process_data * 1000.0) self.statsd_timing("store_data", total_time_store_data * 1000.0) status = self.visit_status() visit_status = OriginVisitStatus( origin=self.origin.url, visit=self.visit.visit, type=self.visit_type, date=now(), status=status, snapshot=self.loaded_snapshot_id, ) self.storage.origin_visit_status_add([visit_status]) success = True with self.statsd_timed( "post_load", tags={"success": success, "status": status} ): self.post_load() except BaseException as e: success = False if isinstance(e, NotFound): status = "not_found" task_status = "uneventful" else: status = "partial" if self.loaded_snapshot_id else "failed" task_status = "failed" self.log.exception( "Loading failure, updating to `%s` status", status, extra={ "swh_task_args": [], "swh_task_kwargs": { "origin": self.origin.url, "lister_name": self.lister_name, "lister_instance_name": self.lister_instance_name, }, }, ) if not isinstance(e, (SystemExit, KeyboardInterrupt)): sentry_sdk.capture_exception() visit_status = OriginVisitStatus( origin=self.origin.url, visit=self.visit.visit, type=self.visit_type, date=now(), status=status, snapshot=self.loaded_snapshot_id, ) self.storage.origin_visit_status_add([visit_status]) with self.statsd_timed( "post_load", tags={"success": success, "status": status} ): self.post_load(success=success) if not isinstance(e, Exception): # e derives from BaseException but not Exception; this is most likely # SystemExit or KeyboardInterrupt, so we should re-raise it. raise return {"status": task_status} finally: with self.statsd_timed( "flush", tags={"success": success, "status": status} ): self.flush() with self.statsd_timed( "cleanup", tags={"success": success, "status": status} ): self.cleanup() return self.load_status() def load_metadata_objects( self, metadata_objects: List[RawExtrinsicMetadata] ) -> None: if not metadata_objects: return authorities = {mo.authority for mo in metadata_objects} self.storage.metadata_authority_add(list(authorities)) fetchers = {mo.fetcher for mo in metadata_objects} self.storage.metadata_fetcher_add(list(fetchers)) self.storage.raw_extrinsic_metadata_add(metadata_objects) def build_extrinsic_origin_metadata(self) -> List[RawExtrinsicMetadata]: """Builds a list of full RawExtrinsicMetadata objects, using a metadata fetcher returned by :func:`get_fetcher_classes`.""" if self.lister_name is None: self.log.debug("lister_not provided, skipping extrinsic origin metadata") return [] assert ( self.lister_instance_name is not None ), "lister_instance_name is None, but lister_name is not" metadata = [] fetcher_classes = get_fetchers_for_lister(self.lister_name) self.statsd_average("metadata_fetchers", len(fetcher_classes)) for cls in fetcher_classes: metadata_fetcher = cls( origin=self.origin, lister_name=self.lister_name, lister_instance_name=self.lister_instance_name, credentials=self.metadata_fetcher_credentials, ) with self.statsd_timed( "fetch_one_metadata", tags={"fetcher": cls.FETCHER_NAME} ): metadata.extend(metadata_fetcher.get_origin_metadata()) if self.parent_origins is None: self.parent_origins = metadata_fetcher.get_parent_origins() self.statsd_average( "metadata_parent_origins", len(self.parent_origins), tags={"fetcher": cls.FETCHER_NAME}, ) self.statsd_average("metadata_objects", len(metadata)) return metadata def statsd_timed(self, name: str, tags: Dict[str, Any] = {}) -> ContextManager: """ Wrapper for :meth:`swh.core.statsd.Statsd.timed`, which uses the standard metric name and tags for loaders. """ return self.statsd.timed( "operation_duration_seconds", tags={"operation": name, **tags} ) def statsd_timing(self, name: str, value: float, tags: Dict[str, Any] = {}) -> None: """ Wrapper for :meth:`swh.core.statsd.Statsd.timing`, which uses the standard metric name and tags for loaders. """ self.statsd.timing( "operation_duration_seconds", value, tags={"operation": name, **tags} ) def statsd_average( self, name: str, value: Union[int, float], tags: Dict[str, Any] = {} ) -> None: """Increments both ``{name}_sum`` (by the ``value``) and ``{name}_count`` (by ``1``), allowing to prometheus to compute the average ``value`` over time.""" self.statsd.increment(f"{name}_sum", value, tags=tags) self.statsd.increment(f"{name}_count", tags=tags) class DVCSLoader(BaseLoader): """This base class is a pattern for dvcs loaders (e.g. git, mercurial). Those loaders are able to load all the data in one go. For example, the loader defined in swh-loader-git :class:`BulkUpdater`. For other loaders (stateful one, (e.g :class:`SWHSvnLoader`), inherit directly from :class:`BaseLoader`. """ def cleanup(self) -> None: """Clean up an eventual state installed for computations.""" pass def has_contents(self) -> bool: """Checks whether we need to load contents""" return True def get_contents(self) -> Iterable[BaseContent]: """Get the contents that need to be loaded""" raise NotImplementedError def has_directories(self) -> bool: """Checks whether we need to load directories""" return True def get_directories(self) -> Iterable[Directory]: """Get the directories that need to be loaded""" raise NotImplementedError def has_revisions(self) -> bool: """Checks whether we need to load revisions""" return True def get_revisions(self) -> Iterable[Revision]: """Get the revisions that need to be loaded""" raise NotImplementedError def has_releases(self) -> bool: """Checks whether we need to load releases""" return True def get_releases(self) -> Iterable[Release]: """Get the releases that need to be loaded""" raise NotImplementedError def get_snapshot(self) -> Snapshot: """Get the snapshot that needs to be loaded""" raise NotImplementedError def eventful(self) -> bool: """Whether the load was eventful""" raise NotImplementedError def store_data(self) -> None: assert self.origin if self.save_data_path: self.save_data() if self.has_contents(): for obj in self.get_contents(): if isinstance(obj, Content): self.storage.content_add([obj]) elif isinstance(obj, SkippedContent): self.storage.skipped_content_add([obj]) else: raise TypeError(f"Unexpected content type: {obj}") if self.has_directories(): for directory in self.get_directories(): self.storage.directory_add([directory]) if self.has_revisions(): for revision in self.get_revisions(): self.storage.revision_add([revision]) if self.has_releases(): for release in self.get_releases(): self.storage.release_add([release]) snapshot = self.get_snapshot() self.storage.snapshot_add([snapshot]) self.flush() self.loaded_snapshot_id = snapshot.id diff --git a/swh/loader/core/tests/test_loader.py b/swh/loader/core/tests/test_loader.py index dacec8b..6633460 100644 --- a/swh/loader/core/tests/test_loader.py +++ b/swh/loader/core/tests/test_loader.py @@ -1,481 +1,505 @@ # Copyright (C) 2018-2021 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU General Public License version 3, or any later version # See top-level LICENSE file for more information import datetime import hashlib import logging import time from unittest.mock import MagicMock, call import pytest -from swh.loader.core.loader import BaseLoader, DVCSLoader +from swh.loader.core.loader import ( + SENTRY_ORIGIN_URL_TAG_NAME, + SENTRY_VISIT_TYPE_TAG_NAME, + BaseLoader, + DVCSLoader, +) from swh.loader.core.metadata_fetchers import MetadataFetcherProtocol from swh.loader.exception import NotFound from swh.loader.tests import assert_last_visit_matches from swh.model.hashutil import hash_to_bytes from swh.model.model import ( MetadataAuthority, MetadataAuthorityType, MetadataFetcher, Origin, RawExtrinsicMetadata, Snapshot, ) import swh.storage.exc ORIGIN = Origin(url="some-url") PARENT_ORIGIN = Origin(url="base-origin-url") METADATA_AUTHORITY = MetadataAuthority( type=MetadataAuthorityType.FORGE, url="http://example.org/" ) REMD = RawExtrinsicMetadata( target=ORIGIN.swhid(), discovery_date=datetime.datetime.now(tz=datetime.timezone.utc), authority=METADATA_AUTHORITY, fetcher=MetadataFetcher( name="test fetcher", version="0.0.1", ), format="test-format", metadata=b'{"foo": "bar"}', ) class DummyLoader: """Base Loader to overload and simplify the base class (technical: to avoid repetition in other *Loader classes)""" visit_type = "git" def __init__(self, storage, *args, **kwargs): super().__init__(storage, ORIGIN.url, *args, **kwargs) def cleanup(self): pass def prepare(self, *args, **kwargs): pass def fetch_data(self): pass def get_snapshot_id(self): return None class DummyDVCSLoader(DummyLoader, DVCSLoader): """DVCS Loader that does nothing in regards to DAG objects.""" def get_contents(self): return [] def get_directories(self): return [] def get_revisions(self): return [] def get_releases(self): return [] def get_snapshot(self): return Snapshot(branches={}) def eventful(self): return False class DummyBaseLoader(DummyLoader, BaseLoader): """Buffered loader will send new data when threshold is reached""" def store_data(self): pass class DummyMetadataFetcher: SUPPORTED_LISTERS = {"fake-forge"} FETCHER_NAME = "fake-forge" def __init__(self, origin, credentials, lister_name, lister_instance_name): pass def get_origin_metadata(self): return [REMD] def get_parent_origins(self): return [] class DummyMetadataFetcherWithFork: SUPPORTED_LISTERS = {"fake-forge"} FETCHER_NAME = "fake-forge" def __init__(self, origin, credentials, lister_name, lister_instance_name): pass def get_origin_metadata(self): return [REMD] def get_parent_origins(self): return [PARENT_ORIGIN] def test_types(): assert isinstance( DummyMetadataFetcher(None, None, None, None), MetadataFetcherProtocol ) assert isinstance( DummyMetadataFetcherWithFork(None, None, None, None), MetadataFetcherProtocol ) def test_base_loader(swh_storage): loader = DummyBaseLoader(swh_storage) result = loader.load() assert result == {"status": "eventful"} def test_base_loader_with_config(swh_storage): loader = DummyBaseLoader(swh_storage, "logger-name") result = loader.load() assert result == {"status": "eventful"} def test_base_loader_with_known_lister_name(swh_storage, mocker): fetcher_cls = MagicMock(wraps=DummyMetadataFetcher) fetcher_cls.SUPPORTED_LISTERS = DummyMetadataFetcher.SUPPORTED_LISTERS fetcher_cls.FETCHER_NAME = "fake-forge" mocker.patch( "swh.loader.core.metadata_fetchers._fetchers", return_value=[fetcher_cls] ) loader = DummyBaseLoader( swh_storage, lister_name="fake-forge", lister_instance_name="" ) statsd_report = mocker.patch.object(loader.statsd, "_report") result = loader.load() assert result == {"status": "eventful"} fetcher_cls.assert_called_once() fetcher_cls.assert_called_once_with( origin=ORIGIN, credentials={}, lister_name="fake-forge", lister_instance_name="", ) assert swh_storage.raw_extrinsic_metadata_get( ORIGIN.swhid(), METADATA_AUTHORITY ).results == [REMD] assert loader.parent_origins == [] assert [ call("metadata_fetchers_sum", "c", 1, {}, 1), call("metadata_fetchers_count", "c", 1, {}, 1), call("metadata_parent_origins_sum", "c", 0, {"fetcher": "fake-forge"}, 1), call("metadata_parent_origins_count", "c", 1, {"fetcher": "fake-forge"}, 1), call("metadata_objects_sum", "c", 1, {}, 1), call("metadata_objects_count", "c", 1, {}, 1), ] == [c for c in statsd_report.mock_calls if "metadata_" in c[1][0]] assert loader.statsd.namespace == "swh_loader" assert loader.statsd.constant_tags == {"visit_type": "git"} def test_base_loader_with_unknown_lister_name(swh_storage, mocker): fetcher_cls = MagicMock(wraps=DummyMetadataFetcher) fetcher_cls.SUPPORTED_LISTERS = DummyMetadataFetcher.SUPPORTED_LISTERS mocker.patch( "swh.loader.core.metadata_fetchers._fetchers", return_value=[fetcher_cls] ) loader = DummyBaseLoader( swh_storage, lister_name="other-lister", lister_instance_name="" ) result = loader.load() assert result == {"status": "eventful"} fetcher_cls.assert_not_called() with pytest.raises(swh.storage.exc.StorageArgumentException): swh_storage.raw_extrinsic_metadata_get(ORIGIN.swhid(), METADATA_AUTHORITY) def test_base_loader_forked_origin(swh_storage, mocker): fetcher_cls = MagicMock(wraps=DummyMetadataFetcherWithFork) fetcher_cls.SUPPORTED_LISTERS = DummyMetadataFetcherWithFork.SUPPORTED_LISTERS fetcher_cls.FETCHER_NAME = "fake-forge" mocker.patch( "swh.loader.core.metadata_fetchers._fetchers", return_value=[fetcher_cls] ) loader = DummyBaseLoader( swh_storage, lister_name="fake-forge", lister_instance_name="" ) statsd_report = mocker.patch.object(loader.statsd, "_report") result = loader.load() assert result == {"status": "eventful"} fetcher_cls.assert_called_once() fetcher_cls.assert_called_once_with( origin=ORIGIN, credentials={}, lister_name="fake-forge", lister_instance_name="", ) assert swh_storage.raw_extrinsic_metadata_get( ORIGIN.swhid(), METADATA_AUTHORITY ).results == [REMD] assert loader.parent_origins == [PARENT_ORIGIN] assert [ call("metadata_fetchers_sum", "c", 1, {}, 1), call("metadata_fetchers_count", "c", 1, {}, 1), call("metadata_parent_origins_sum", "c", 1, {"fetcher": "fake-forge"}, 1), call("metadata_parent_origins_count", "c", 1, {"fetcher": "fake-forge"}, 1), call("metadata_objects_sum", "c", 1, {}, 1), call("metadata_objects_count", "c", 1, {}, 1), ] == [c for c in statsd_report.mock_calls if "metadata_" in c[1][0]] assert loader.statsd.namespace == "swh_loader" assert loader.statsd.constant_tags == {"visit_type": "git"} def test_base_loader_post_load_raise(swh_storage, mocker): loader = DummyBaseLoader(swh_storage) post_load = mocker.patch.object(loader, "post_load") # raise exception in post_load when success is True def post_load_method(*args, success=True): if success: raise Exception("Error in post_load") post_load.side_effect = post_load_method result = loader.load() assert result == {"status": "failed"} # ensure post_load has been called twice, once with success to True and # once with success to False as the first post_load call raised exception assert post_load.call_args_list == [mocker.call(), mocker.call(success=False)] def test_dvcs_loader(swh_storage): loader = DummyDVCSLoader(swh_storage) result = loader.load() assert result == {"status": "eventful"} def test_dvcs_loader_with_config(swh_storage): loader = DummyDVCSLoader(swh_storage, "another-logger") result = loader.load() assert result == {"status": "eventful"} def test_loader_logger_default_name(swh_storage): loader = DummyBaseLoader(swh_storage) assert isinstance(loader.log, logging.Logger) assert loader.log.name == "swh.loader.core.tests.test_loader.DummyBaseLoader" loader = DummyDVCSLoader(swh_storage) assert isinstance(loader.log, logging.Logger) assert loader.log.name == "swh.loader.core.tests.test_loader.DummyDVCSLoader" def test_loader_logger_with_name(swh_storage): loader = DummyBaseLoader(swh_storage, "some.logger.name") assert isinstance(loader.log, logging.Logger) assert loader.log.name == "some.logger.name" def test_loader_save_data_path(swh_storage, tmp_path): loader = DummyBaseLoader(swh_storage, "some.logger.name.1", save_data_path=tmp_path) url = "http://bitbucket.org/something" loader.origin = Origin(url=url) loader.visit_date = datetime.datetime(year=2019, month=10, day=1) hash_url = hashlib.sha1(url.encode("utf-8")).hexdigest() expected_save_path = "%s/sha1:%s/%s/2019" % (str(tmp_path), hash_url[0:2], hash_url) save_path = loader.get_save_data_path() assert save_path == expected_save_path def _check_load_failure(caplog, loader, exc_class, exc_text, status="partial"): """Check whether a failed load properly logged its exception, and that the snapshot didn't get referenced in storage""" assert isinstance(loader, DVCSLoader) # was implicit so far for record in caplog.records: if record.levelname != "ERROR": continue assert "Loading failure" in record.message assert record.exc_info exc = record.exc_info[1] assert isinstance(exc, exc_class) assert exc_text in exc.args[0] # Check that the get_snapshot operation would have succeeded assert loader.get_snapshot() is not None # And confirm that the visit doesn't reference a snapshot visit = assert_last_visit_matches(loader.storage, ORIGIN.url, status) if status != "partial": assert visit.snapshot is None # But that the snapshot didn't get loaded assert loader.loaded_snapshot_id is None @pytest.mark.parametrize("success", [True, False]) def test_loader_timings(swh_storage, mocker, success): current_time = time.time() mocker.patch("time.monotonic", side_effect=lambda: current_time) mocker.patch("swh.core.statsd.monotonic", side_effect=lambda: current_time) runtimes = { "pre_cleanup": 2.0, "build_extrinsic_origin_metadata": 3.0, "prepare": 5.0, "fetch_data": 7.0, "process_data": 11.0, "store_data": 13.0, "post_load": 17.0, "flush": 23.0, "cleanup": 27.0, } class TimedLoader(BaseLoader): visit_type = "my-visit-type" def __getattribute__(self, method_name): if method_name == "visit_status" and not success: def crashy(): raise Exception("oh no") return crashy if method_name not in runtimes: return super().__getattribute__(method_name) def meth(*args, **kwargs): nonlocal current_time current_time += runtimes[method_name] return meth loader = TimedLoader(swh_storage, origin_url="http://example.org/hello.git") statsd_report = mocker.patch.object(loader.statsd, "_report") loader.load() if success: expected_tags = { "post_load": {"success": True, "status": "full"}, "flush": {"success": True, "status": "full"}, "cleanup": {"success": True, "status": "full"}, } else: expected_tags = { "post_load": {"success": False, "status": "failed"}, "flush": {"success": False, "status": "failed"}, "cleanup": {"success": False, "status": "failed"}, } # note that this is a list equality, so order of entries in 'runtimes' matters. # This is not perfect, but call() objects are not hashable so it's simpler this way, # even if not perfect. assert statsd_report.mock_calls == [ call( "operation_duration_seconds", "ms", value * 1000, {"operation": key, **expected_tags.get(key, {})}, 1, ) for (key, value) in runtimes.items() ] assert loader.statsd.namespace == "swh_loader" assert loader.statsd.constant_tags == {"visit_type": "my-visit-type"} class DummyDVCSLoaderExc(DummyDVCSLoader): """A loader which raises an exception when loading some contents""" def get_contents(self): raise RuntimeError("Failed to get contents!") def test_dvcs_loader_exc_partial_visit(swh_storage, caplog): logger_name = "dvcsloaderexc" caplog.set_level(logging.ERROR, logger=logger_name) loader = DummyDVCSLoaderExc(swh_storage, logging_class=logger_name) # fake the loading ending up in a snapshot loader.loaded_snapshot_id = hash_to_bytes( "9e4dd2b40d1b46b70917c0949aa2195c823a648e" ) result = loader.load() # loading failed assert result == {"status": "failed"} # still resulted in a partial visit with a snapshot (somehow) _check_load_failure( caplog, loader, RuntimeError, "Failed to get contents!", ) class BrokenStorageProxy: def __init__(self, storage): self.storage = storage def __getattr__(self, attr): return getattr(self.storage, attr) def snapshot_add(self, snapshots): raise RuntimeError("Failed to add snapshot!") class DummyDVCSLoaderStorageExc(DummyDVCSLoader): """A loader which raises an exception when loading some contents""" def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.storage = BrokenStorageProxy(self.storage) def test_dvcs_loader_storage_exc_failed_visit(swh_storage, caplog): logger_name = "dvcsloaderexc" caplog.set_level(logging.ERROR, logger=logger_name) loader = DummyDVCSLoaderStorageExc(swh_storage, logging_class=logger_name) result = loader.load() assert result == {"status": "failed"} _check_load_failure( caplog, loader, RuntimeError, "Failed to add snapshot!", status="failed" ) class DummyDVCSLoaderNotFound(DummyDVCSLoader, BaseLoader): """A loader which raises a not_found exception during the prepare method call""" def prepare(*args, **kwargs): raise NotFound("Unknown origin!") def load_status(self): return { "status": "uneventful", } def test_loader_not_found(swh_storage, caplog): loader = DummyDVCSLoaderNotFound(swh_storage) result = loader.load() assert result == {"status": "uneventful"} _check_load_failure(caplog, loader, NotFound, "Unknown origin!", status="not_found") + + +class DummyLoaderWithError(DummyBaseLoader): + def prepare(self, *args, **kwargs): + raise Exception("error") + + +class DummyDVCSLoaderWithError(DummyDVCSLoader, BaseLoader): + def prepare(self, *args, **kwargs): + raise Exception("error") + + +@pytest.mark.parametrize("loader_cls", [DummyLoaderWithError, DummyDVCSLoaderWithError]) +def test_loader_sentry_tags_on_error(swh_storage, sentry_events, loader_cls): + loader = loader_cls(swh_storage) + loader.load() + sentry_tags = sentry_events[0]["tags"] + assert sentry_tags.get(SENTRY_ORIGIN_URL_TAG_NAME) == ORIGIN.url + assert sentry_tags.get(SENTRY_VISIT_TYPE_TAG_NAME) == DummyLoader.visit_type diff --git a/swh/loader/package/pubdev/loader.py b/swh/loader/package/pubdev/loader.py index 2a0a944..bcce138 100644 --- a/swh/loader/package/pubdev/loader.py +++ b/swh/loader/package/pubdev/loader.py @@ -1,194 +1,194 @@ # Copyright (C) 2022 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU General Public License version 3, or any later version # See top-level LICENSE file for more information -from distutils.version import LooseVersion import json from pathlib import Path from typing import Any, Dict, Iterator, Optional, Sequence, Tuple import attr +from packaging.version import parse as parse_version import yaml from swh.loader.package.loader import BasePackageInfo, PackageLoader from swh.loader.package.utils import ( EMPTY_AUTHOR, Person, api_info, cached_method, release_name, ) from swh.model.model import ObjectType, Release, Sha1Git, TimestampWithTimezone from swh.storage.interface import StorageInterface @attr.s class PubDevPackageInfo(BasePackageInfo): name = attr.ib(type=str) """Name of the package""" version = attr.ib(type=str) """Current version""" last_modified = attr.ib(type=str) """Last modified date as release date""" author = attr.ib(type=Person) """Author""" description = attr.ib(type=str) """Description""" def extract_intrinsic_metadata(dir_path: Path) -> Dict[str, Any]: """Extract intrinsic metadata from pubspec.yaml file at dir_path. Each pub.dev package version has a pubspec.yaml file at the root of the archive. See https://dart.dev/tools/pub/pubspec for pubspec specifications. Args: dir_path: A directory on disk where a pubspec.yaml must be present Returns: A dict mapping from yaml parser """ pubspec_path = dir_path / "pubspec.yaml" return yaml.safe_load(pubspec_path.read_text()) class PubDevLoader(PackageLoader[PubDevPackageInfo]): visit_type = "pubdev" PUBDEV_BASE_URL = "https://pub.dev/" def __init__( self, storage: StorageInterface, url: str, **kwargs, ): super().__init__(storage=storage, url=url, **kwargs) self.url = url assert url.startswith(self.PUBDEV_BASE_URL) self.package_info_url = url.replace( self.PUBDEV_BASE_URL, f"{self.PUBDEV_BASE_URL}api/" ) def _raw_info(self) -> bytes: return api_info(self.package_info_url) @cached_method def info(self) -> Dict: """Return the project metadata information (fetched from pub.dev registry)""" # Use strict=False in order to correctly manage case where \n is present in a string info = json.loads(self._raw_info(), strict=False) # Arrange versions list as a new dict with `version` as key versions = {v["version"]: v for v in info["versions"]} info["versions"] = versions return info def get_versions(self) -> Sequence[str]: """Get all released versions of a PubDev package Returns: A sequence of versions Example:: ["0.1.1", "0.10.2"] """ versions = list(self.info()["versions"].keys()) - versions.sort(key=LooseVersion) + versions.sort(key=parse_version) return versions def get_default_version(self) -> str: """Get the newest release version of a PubDev package Returns: A string representing a version Example:: "0.1.2" """ latest = self.info()["latest"] return latest["version"] def get_package_info(self, version: str) -> Iterator[Tuple[str, PubDevPackageInfo]]: """Get release name and package information from version Package info comes from extrinsic metadata (from self.info()) Args: version: Package version (e.g: "0.1.0") Returns: Iterator of tuple (release_name, p_info) """ v = self.info()["versions"][version] assert v["version"] == version url = v["archive_url"] name = v["pubspec"]["name"] filename = f"{name}-{version}.tar.gz" last_modified = v["published"] if "authors" in v["pubspec"]: # TODO: here we have a list of author, see T3887 author = Person.from_fullname(v["pubspec"]["authors"][0].encode()) elif "author" in v["pubspec"] and v["pubspec"]["author"] is not None: author = Person.from_fullname(v["pubspec"]["author"].encode()) else: author = EMPTY_AUTHOR description = v["pubspec"]["description"] p_info = PubDevPackageInfo( name=name, filename=filename, url=url, version=version, last_modified=last_modified, author=author, description=description, ) yield release_name(version), p_info def build_release( self, p_info: PubDevPackageInfo, uncompressed_path: str, directory: Sha1Git ) -> Optional[Release]: # Extract intrinsic metadata from uncompressed_path/pubspec.yaml intrinsic_metadata = extract_intrinsic_metadata(Path(uncompressed_path)) name: str = intrinsic_metadata["name"] version: str = intrinsic_metadata["version"] assert version == p_info.version # author from intrinsic_metadata should not take precedence over the one # returned by the api, see https://dart.dev/tools/pub/pubspec#authorauthors author: Person = p_info.author if "description" in intrinsic_metadata and intrinsic_metadata["description"]: description = intrinsic_metadata["description"] else: description = p_info.description message = ( f"Synthetic release for pub.dev source package {name} " f"version {version}\n\n" f"{description}\n" ) return Release( name=version.encode(), author=author, date=TimestampWithTimezone.from_iso8601(p_info.last_modified), message=message.encode(), target_type=ObjectType.DIRECTORY, target=directory, synthetic=True, ) diff --git a/swh/loader/package/pubdev/tests/data/fake_pubdev.sh b/swh/loader/package/pubdev/tests/data/fake_pubdev.sh index c4c33dd..5d2f424 100644 --- a/swh/loader/package/pubdev/tests/data/fake_pubdev.sh +++ b/swh/loader/package/pubdev/tests/data/fake_pubdev.sh @@ -1,194 +1,176 @@ #!/usr/bin/env bash -# Script to generate fake pub.dev http api response and fake Dart or FLutter packages archives as .tar.gz. +# Script to generate fake Dart or FLutter packages archives as .tar.gz. set -euo pipefail # Create directories readonly TMP=tmp_dir/pubdev -readonly BASE_API=https_pub.dev readonly BASE_ARCHIVES=https_pub.dartlang.org mkdir -p $TMP -mkdir -p $BASE_API mkdir -p $BASE_ARCHIVES -# http api response as json -echo -e '''{"name":"authentication","latest":{"version":"0.0.1","pubspec":{"name":"authentication","description":"Persistent user authentication for Flutter with optional backend API integration.","version":"0.0.1","author":null,"homepage":null,"environment":{"sdk":">=2.7.0 <3.0.0","flutter":">=1.17.0 <2.0.0"},"dependencies":{"flutter":{"sdk":"flutter"}},"dev_dependencies":{"flutter_test":{"sdk":"flutter"}},"flutter":{"plugin":{"platforms":{"some_platform":{"pluginClass":"somePluginClass"}}}}},"archive_url":"https://pub.dartlang.org/packages/authentication/versions/0.0.1.tar.gz","archive_sha256":"0179334b346cb67e4e6e3c905e5cc5c8e488a45ebd99fd2be3a7e0476d620d99","published":"2020-08-13T04:53:34.134687Z"},"versions":[{"version":"0.0.1","pubspec":{"name":"authentication","description":"Persistent user authentication for Flutter with optional backend API integration.","version":"0.0.1","author":null,"homepage":null,"environment":{"sdk":">=2.7.0 <3.0.0","flutter":">=1.17.0 <2.0.0"},"dependencies":{"flutter":{"sdk":"flutter"}},"dev_dependencies":{"flutter_test":{"sdk":"flutter"}},"flutter":{"plugin":{"platforms":{"some_platform":{"pluginClass":"somePluginClass"}}}}},"archive_url":"https://pub.dartlang.org/packages/authentication/versions/0.0.1.tar.gz","archive_sha256":"0179334b346cb67e4e6e3c905e5cc5c8e488a45ebd99fd2be3a7e0476d620d99","published":"2020-08-13T04:53:34.134687Z"}]} -''' > $BASE_API/api_packages_authentication - -echo -e '''{"name":"Autolinker","latest":{"version":"0.1.1","pubspec":{"version":"0.1.1","homepage":"https://github.com/hackcave","description":"Port of Autolinker.js to dart","name":"Autolinker","author":"hackcave "},"archive_url":"https://pub.dartlang.org/packages/Autolinker/versions/0.1.1.tar.gz","published":"2014-12-24T22:34:02.534090Z"},"versions":[{"version":"0.1.1","pubspec":{"version":"0.1.1","homepage":"https://github.com/hackcave","description":"Port of Autolinker.js to dart","name":"Autolinker","author":"hackcave "},"archive_url":"https://pub.dartlang.org/packages/Autolinker/versions/0.1.1.tar.gz","published":"2014-12-24T22:34:02.534090Z"}]} -''' > ${BASE_API}/api_packages_Autolinker - -echo -e '''{"name":"bezier","latest":{"version":"1.1.5","pubspec":{"name":"bezier","version":"1.1.5","authors":["Aaron Barrett ","Isaac Barrett "],"description":"A 2D Bézier curve math library. Based heavily on the work of @TheRealPomax .\nLive examples can be found at .","homepage":"https://github.com/aab29/bezier.dart","environment":{"sdk":">=2.0.0 <3.0.0"},"dependencies":{"vector_math":"^2.0.0"},"dev_dependencies":{"test":"^1.0.0"}},"archive_url":"https://pub.dartlang.org/packages/bezier/versions/1.1.5.tar.gz","archive_sha256":"cc5da2fa927b5d347550f78d456cd984b7df78a7f0405119cdab12111e2f9ee8","published":"2019-12-22T03:17:30.805225Z"},"versions":[{"version":"1.1.5","pubspec":{"name":"bezier","version":"1.1.5","authors":["Aaron Barrett ","Isaac Barrett "],"description":"A 2D Bézier curve math library. Based heavily on the work of @TheRealPomax .\nLive examples can be found at .","homepage":"https://github.com/aab29/bezier.dart","environment":{"sdk":">=2.0.0 <3.0.0"},"dependencies":{"vector_math":"^2.0.0"},"dev_dependencies":{"test":"^1.0.0"}},"archive_url":"https://pub.dartlang.org/packages/bezier/versions/1.1.5.tar.gz","archive_sha256":"cc5da2fa927b5d347550f78d456cd984b7df78a7f0405119cdab12111e2f9ee8","published":"2019-12-22T03:17:30.805225Z"}]} -''' > ${BASE_API}/api_packages_bezier - -echo -e '''{"name":"pdf","latest":{"version":"3.8.2","pubspec":{"name":"pdf","description":"A pdf producer for Dart. It can create pdf files for both web or flutter.","homepage":"https://github.com/DavBfr/dart_pdf/tree/master/pdf","repository":"https://github.com/DavBfr/dart_pdf","issue_tracker":"https://github.com/DavBfr/dart_pdf/issues","version":"3.8.2","environment":{"sdk":">=2.12.0 <3.0.0"},"dependencies":{"archive":"^3.1.0","barcode":">=2.2.0 <3.0.0","crypto":"^3.0.0","image":">=3.0.1 <4.0.0","meta":">=1.3.0 <2.0.0","path_parsing":">=0.2.0 <2.0.0","vector_math":"^2.1.0","xml":">=5.1.0 <7.0.0"},"dev_dependencies":{"flutter_lints":"^1.0.4","test":">=1.16.0 <2.0.0"}},"archive_url":"https://pub.dartlang.org/packages/pdf/versions/3.8.2.tar.gz","published":"2022-07-25T11:38:25.983876Z"},"versions":[{"version":"1.0.0","pubspec":{"version":"1.0.0","name":"pdf","dependencies":{"ttf_parser":"^1.0.0","vector_math":"^2.0.7","meta":"^1.1.5"},"author":"David PHAM-VAN ","description":"A pdf producer for Dart","homepage":"https://github.com/davbfr/dart_pdf","environment":{"sdk":">=1.8.0 <2.0.0"},"dev_dependencies":{"test":"any"}},"archive_url":"https://pub.dartlang.org/packages/pdf/versions/1.0.0.tar.gz","published":"2018-07-16T21:12:28.894137Z"},{"version":"3.8.2","pubspec":{"name":"pdf","description":"A pdf producer for Dart. It can create pdf files for both web or flutter.","homepage":"https://github.com/DavBfr/dart_pdf/tree/master/pdf","repository":"https://github.com/DavBfr/dart_pdf","issue_tracker":"https://github.com/DavBfr/dart_pdf/issues","version":"3.8.2","environment":{"sdk":">=2.12.0 <3.0.0"},"dependencies":{"archive":"^3.1.0","barcode":">=2.2.0 <3.0.0","crypto":"^3.0.0","image":">=3.0.1 <4.0.0","meta":">=1.3.0 <2.0.0","path_parsing":">=0.2.0 <2.0.0","vector_math":"^2.1.0","xml":">=5.1.0 <7.0.0"},"dev_dependencies":{"flutter_lints":"^1.0.4","test":">=1.16.0 <2.0.0"}},"archive_url":"https://pub.dartlang.org/packages/pdf/versions/3.8.2.tar.gz","published":"2022-07-25T11:38:25.983876Z"}]} -''' > ${BASE_API}/api_packages_pdf - -echo -e '''{"name":"abstract_io","latest":{"version":"0.1.2+6","pubspec":{"name":"abstract_io","description":"Abstract IO is designed to simplify and generalize saving data both localy and externaly","version":"0.1.2+6","author":"Anders Groeschel","repository":"https://github.com/AndersGroeschel/abstract_io","homepage":"https://github.com/AndersGroeschel/abstract_io","environment":{"sdk":">=2.7.0 <3.0.0"},"dependencies":{"flutter":{"sdk":"flutter"}}},"archive_url":"https://pub.dartlang.org/packages/abstract_io/versions/0.1.2%2B6.tar.gz","archive_sha256":"9557fd384730d92a046cfccdff9625f2d646657219d5a0e447cb7eb0fdf90f18","published":"2020-08-03T21:31:05.764846Z"},"versions":[{"version":"0.1.2+4","pubspec":{"name":"abstract_io","description":"Abstract IO is designed to simplify and generalize saving data both localy and externaly","version":"0.1.2+4","author":"Anders Groeschel","repository":"https://github.com/AndersGroeschel/abstract_io","homepage":"https://github.com/AndersGroeschel/abstract_io","environment":{"sdk":">=2.7.0 <3.0.0"},"dependencies":{"flutter":{"sdk":"flutter"}}},"archive_url":"https://pub.dartlang.org/packages/abstract_io/versions/0.1.2%2B4.tar.gz","archive_sha256":"df687ff2a92774db04a28167ccddbfe9c2fc1ea63c6ae05c3236552fe350bb68","published":"2020-08-03T20:14:38.116237Z"},{"version":"0.1.2+5","pubspec":{"name":"abstract_io","description":"Abstract IO is designed to simplify and generalize saving data both localy and externaly","version":"0.1.2+5","author":"Anders Groeschel","repository":"https://github.com/AndersGroeschel/abstract_io","homepage":"https://github.com/AndersGroeschel/abstract_io","environment":{"sdk":">=2.7.0 <3.0.0"},"dependencies":{"flutter":{"sdk":"flutter"}}},"archive_url":"https://pub.dartlang.org/packages/abstract_io/versions/0.1.2%2B5.tar.gz","archive_sha256":"fc9199c2f9879d3c0d140c05a2f8c537561af256d98d209b4ee102e8107ec2b9","published":"2020-08-03T21:09:20.329418Z"},{"version":"0.1.2+6","pubspec":{"name":"abstract_io","description":"Abstract IO is designed to simplify and generalize saving data both localy and externaly","version":"0.1.2+6","author":"Anders Groeschel","repository":"https://github.com/AndersGroeschel/abstract_io","homepage":"https://github.com/AndersGroeschel/abstract_io","environment":{"sdk":">=2.7.0 <3.0.0"},"dependencies":{"flutter":{"sdk":"flutter"}}},"archive_url":"https://pub.dartlang.org/packages/abstract_io/versions/0.1.2%2B6.tar.gz","archive_sha256":"9557fd384730d92a046cfccdff9625f2d646657219d5a0e447cb7eb0fdf90f18","published":"2020-08-03T21:31:05.764846Z"}]} -''' > ${BASE_API}/api_packages_abstract_io - # Dart package a pubspec.yaml file at thier root. Generate some of them. mkdir -p ${TMP}/packages_authentication_versions_0.0.1 echo -e '''name: authentication description: Persistent user authentication for Flutter with optional backend API integration. version: 0.0.1 author: homepage: environment: sdk: ">=2.7.0 <3.0.0" flutter: ">=1.17.0 <2.0.0" dependencies: flutter: sdk: flutter dev_dependencies: flutter_test: sdk: flutter # For information on the generic Dart part of this file, see the # following page: https://dart.dev/tools/pub/pubspec # The following section is specific to Flutter. flutter: # This section identifies this Flutter project as a plugin project. # The 'pluginClass' and Android 'package' identifiers should not ordinarily # be modified. They are used by the tooling to maintain consistency when # adding or updating assets for this project. plugin: platforms: # This plugin project was generated without specifying any # platforms with the `--platform` argument. If you see the `fake_platform` map below, remove it and # then add platforms following the instruction here: # https://flutter.dev/docs/development/packages-and-plugins/developing-packages#plugin-platforms # ------------------- some_platform: pluginClass: somePluginClass # ------------------- # To add assets to your plugin package, add an assets section, like this: # assets: # - images/a_dot_burr.jpeg # - images/a_dot_ham.jpeg # # For details regarding assets in packages, see # https://flutter.dev/assets-and-images/#from-packages # # An image asset can refer to one or more resolution-specific "variants", see # https://flutter.dev/assets-and-images/#resolution-aware. # To add custom fonts to your plugin package, add a fonts section here, # in this "flutter" section. Each entry in this list should have a # "family" key with the font family name, and a "fonts" key with a # list giving the asset and other descriptors for the font. For # example: # fonts: # - family: Schyler # fonts: # - asset: fonts/Schyler-Regular.ttf # - asset: fonts/Schyler-Italic.ttf # style: italic # - family: Trajan Pro # fonts: # - asset: fonts/TrajanPro.ttf # - asset: fonts/TrajanPro_Bold.ttf # weight: 700 # # For details regarding fonts in packages, see # https://flutter.dev/custom-fonts/#from-packages ''' > ${TMP}/packages_authentication_versions_0.0.1/pubspec.yaml mkdir -p ${TMP}/packages_autolinker_versions_0.1.1 echo -e '''name: Autolinker version: 0.1.1 author: hackcave homepage: https://github.com/hackcave description: Port of Autolinker.js to dart ''' > ${TMP}/packages_autolinker_versions_0.1.1/pubspec.yaml mkdir -p ${TMP}/packages_bezier_versions_1.1.5 echo -e '''name: bezier version: 1.1.5 authors: - Aaron Barrett - Isaac Barrett description: >- A 2D Bézier curve math library. Based heavily on the work of @TheRealPomax . Live examples can be found at . homepage: https://github.com/aab29/bezier.dart environment: sdk: ">=2.0.0 <3.0.0" dependencies: vector_math: ^2.0.0 dev_dependencies: test: ^1.0.0 ''' > ${TMP}/packages_bezier_versions_1.1.5/pubspec.yaml mkdir -p ${TMP}/packages_pdf_versions_1.0.0 echo -e '''name: pdf author: David PHAM-VAN description: A pdf producer for Dart homepage: https://github.com/davbfr/dart_pdf version: 1.0.0 environment: sdk: ">=1.8.0 <2.0.0" dependencies: meta: "^1.1.5" ttf_parser: "^1.0.0" vector_math: "^2.0.7" dev_dependencies: test: any ''' > ${TMP}/packages_pdf_versions_1.0.0/pubspec.yaml mkdir -p ${TMP}/packages_pdf_versions_3.8.2 echo -e '''name: pdf description: A pdf producer for Dart. It can create pdf files for both web or flutter. homepage: https://github.com/DavBfr/dart_pdf/tree/master/pdf repository: https://github.com/DavBfr/dart_pdf issue_tracker: https://github.com/DavBfr/dart_pdf/issues version: 3.8.2 environment: sdk: ">=2.12.0 <3.0.0" dependencies: archive: ^3.1.0 barcode: ">=2.2.0 <3.0.0" crypto: ^3.0.0 image: ">=3.0.1 <4.0.0" meta: ">=1.3.0 <2.0.0" path_parsing: ">=0.2.0 <2.0.0" vector_math: ^2.1.0 xml: ">=5.1.0 <7.0.0" dev_dependencies: flutter_lints: ^1.0.4 test: ">=1.16.0 <2.0.0" ''' > ${TMP}/packages_pdf_versions_3.8.2/pubspec.yaml cd $TMP tar -czf packages_authentication_versions_0.0.1.tar.gz -C packages_authentication_versions_0.0.1 . tar -czf packages_Autolinker_versions_0.1.1.tar.gz -C packages_autolinker_versions_0.1.1 . tar -czf packages_bezier_versions_1.1.5.tar.gz -C packages_bezier_versions_1.1.5 . tar -czf packages_pdf_versions_1.0.0.tar.gz -C packages_pdf_versions_1.0.0 . tar -czf packages_pdf_versions_3.8.2.tar.gz -C packages_pdf_versions_3.8.2 . # Move .tar.gz archives to a servable directory mv *.tar.gz ../../$BASE_ARCHIVES # Clean up removing tmp_dir cd ../../ rm -r tmp_dir/ diff --git a/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_Autolinker_versions_0.1.1.tar.gz b/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_Autolinker_versions_0.1.1.tar.gz index 5cdf2dd..ae42567 100644 Binary files a/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_Autolinker_versions_0.1.1.tar.gz and b/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_Autolinker_versions_0.1.1.tar.gz differ diff --git a/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_authentication_versions_0.0.1.tar.gz b/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_authentication_versions_0.0.1.tar.gz index 4338014..953eef0 100644 Binary files a/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_authentication_versions_0.0.1.tar.gz and b/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_authentication_versions_0.0.1.tar.gz differ diff --git a/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_bezier_versions_1.1.5.tar.gz b/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_bezier_versions_1.1.5.tar.gz index 5a92354..40498cb 100644 Binary files a/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_bezier_versions_1.1.5.tar.gz and b/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_bezier_versions_1.1.5.tar.gz differ diff --git a/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_pdf_versions_1.0.0.tar.gz b/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_pdf_versions_1.0.0.tar.gz index d30d19f..eace6bc 100644 Binary files a/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_pdf_versions_1.0.0.tar.gz and b/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_pdf_versions_1.0.0.tar.gz differ diff --git a/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_pdf_versions_3.8.2.tar.gz b/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_pdf_versions_3.8.2.tar.gz index 45e37e8..c05ec5b 100644 Binary files a/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_pdf_versions_3.8.2.tar.gz and b/swh/loader/package/pubdev/tests/data/https_pub.dartlang.org/packages_pdf_versions_3.8.2.tar.gz differ diff --git a/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_Autolinker b/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_Autolinker index b60f1d8..824af41 100644 --- a/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_Autolinker +++ b/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_Autolinker @@ -1,29 +1,29 @@ { "name": "Autolinker", "latest": { "version": "0.1.1", "pubspec": { "version": "0.1.1", "homepage": "https://github.com/hackcave", "description": "Port of Autolinker.js to dart", "name": "Autolinker", "author": "hackcave " }, "archive_url": "https://pub.dartlang.org/packages/Autolinker/versions/0.1.1.tar.gz", "published": "2014-12-24T22:34:02.534090Z" }, "versions": [ { "version": "0.1.1", "pubspec": { "version": "0.1.1", "homepage": "https://github.com/hackcave", "description": "Port of Autolinker.js to dart", "name": "Autolinker", "author": "hackcave " }, "archive_url": "https://pub.dartlang.org/packages/Autolinker/versions/0.1.1.tar.gz", "published": "2014-12-24T22:34:02.534090Z" } ] -} \ No newline at end of file +} diff --git a/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_abstract_io b/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_abstract_io index 1d00f61..fdd3b7a 100644 --- a/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_abstract_io +++ b/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_abstract_io @@ -1,93 +1,93 @@ { "name": "abstract_io", "latest": { "version": "0.1.2+6", "pubspec": { "name": "abstract_io", "description": "Abstract IO is designed to simplify and generalize saving data both localy and externaly", "version": "0.1.2+6", "author": "Anders Groeschel", "repository": "https://github.com/AndersGroeschel/abstract_io", "homepage": "https://github.com/AndersGroeschel/abstract_io", "environment": { "sdk": ">=2.7.0 <3.0.0" }, "dependencies": { "flutter": { "sdk": "flutter" } } }, "archive_url": "https://pub.dartlang.org/packages/abstract_io/versions/0.1.2%2B6.tar.gz", "archive_sha256": "9557fd384730d92a046cfccdff9625f2d646657219d5a0e447cb7eb0fdf90f18", "published": "2020-08-03T21:31:05.764846Z" }, "versions": [ { "version": "0.1.2+4", "pubspec": { "name": "abstract_io", "description": "Abstract IO is designed to simplify and generalize saving data both localy and externaly", "version": "0.1.2+4", "author": "Anders Groeschel", "repository": "https://github.com/AndersGroeschel/abstract_io", "homepage": "https://github.com/AndersGroeschel/abstract_io", "environment": { "sdk": ">=2.7.0 <3.0.0" }, "dependencies": { "flutter": { "sdk": "flutter" } } }, "archive_url": "https://pub.dartlang.org/packages/abstract_io/versions/0.1.2%2B4.tar.gz", "archive_sha256": "df687ff2a92774db04a28167ccddbfe9c2fc1ea63c6ae05c3236552fe350bb68", "published": "2020-08-03T20:14:38.116237Z" }, { "version": "0.1.2+5", "pubspec": { "name": "abstract_io", "description": "Abstract IO is designed to simplify and generalize saving data both localy and externaly", "version": "0.1.2+5", "author": "Anders Groeschel", "repository": "https://github.com/AndersGroeschel/abstract_io", "homepage": "https://github.com/AndersGroeschel/abstract_io", "environment": { "sdk": ">=2.7.0 <3.0.0" }, "dependencies": { "flutter": { "sdk": "flutter" } } }, "archive_url": "https://pub.dartlang.org/packages/abstract_io/versions/0.1.2%2B5.tar.gz", "archive_sha256": "fc9199c2f9879d3c0d140c05a2f8c537561af256d98d209b4ee102e8107ec2b9", "published": "2020-08-03T21:09:20.329418Z" }, { "version": "0.1.2+6", "pubspec": { "name": "abstract_io", "description": "Abstract IO is designed to simplify and generalize saving data both localy and externaly", "version": "0.1.2+6", "author": "Anders Groeschel", "repository": "https://github.com/AndersGroeschel/abstract_io", "homepage": "https://github.com/AndersGroeschel/abstract_io", "environment": { "sdk": ">=2.7.0 <3.0.0" }, "dependencies": { "flutter": { "sdk": "flutter" } } }, "archive_url": "https://pub.dartlang.org/packages/abstract_io/versions/0.1.2%2B6.tar.gz", "archive_sha256": "9557fd384730d92a046cfccdff9625f2d646657219d5a0e447cb7eb0fdf90f18", "published": "2020-08-03T21:31:05.764846Z" } ] -} \ No newline at end of file +} diff --git a/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_audio_manager b/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_audio_manager new file mode 100644 index 0000000..c3e25fd --- /dev/null +++ b/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_audio_manager @@ -0,0 +1,1637 @@ +{ + "name": "audio_manager", + "latest": { + "version": "0.8.2", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.8.2", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.12.0 <3.0.0", + "flutter": ">=1.20.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + }, + "flutter_web_plugins": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + }, + "web": { + "pluginClass": "AudioManagerPlugin", + "fileName": "audio_manager_web.dart" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.8.2.tar.gz", + "archive_sha256": "6f26b61cc39e258619d0e1bc34500bb7f3926dcc4b6a8c7ea31b67c62381c931", + "published": "2021-10-05T11:25:10.278136Z" + }, + "versions": [ + { + "version": "0.0.1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.0.1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "androidPackage": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.0.1.tar.gz", + "archive_sha256": "ba1e160e0164bf0c3542ec4c337ba93c177b0c656d7fe0dd93e9353b134c3d82", + "published": "2020-01-19T04:03:08.408868Z" + }, + { + "version": "0.0.2", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.0.2", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "androidPackage": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.0.2.tar.gz", + "archive_sha256": "4aec6c5a7473fd9f4778037c390484a0d59d4fa7e5d2ea50e1700524394ecd41", + "published": "2020-01-20T08:17:04.286167Z" + }, + { + "version": "0.1.1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.1.1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "androidPackage": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.1.1.tar.gz", + "archive_sha256": "9b6cc5524fa4ffa946d5ef133497f0bde004cc8ef0875b66c17b36462ade2ae3", + "published": "2020-01-20T08:44:15.151981Z" + }, + { + "version": "0.1.2", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.1.2", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "androidPackage": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.1.2.tar.gz", + "archive_sha256": "20bbc50d15e3ba9d1d9ebe07813b1504d32118f6bd63f06ca557ac657c048dd3", + "published": "2020-01-20T10:10:05.679374Z" + }, + { + "version": "0.1.3", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.1.3", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "androidPackage": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.1.3.tar.gz", + "archive_sha256": "57d2d9ef50edbc74a0d6d23e64c65d25768e7af8c763c6636215651447e261a3", + "published": "2020-01-21T03:26:07.360220Z" + }, + { + "version": "0.1.4", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.1.4", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.1.4.tar.gz", + "archive_sha256": "53ef60933e9c61ac184f7fea38f54c8354eb74dd08f20bfc661eb9f8ae8f84de", + "published": "2020-01-28T12:49:42.158959Z" + }, + { + "version": "0.1.5", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.1.5", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.1.5.tar.gz", + "archive_sha256": "936481a34a3d8eeedcdd26689281ab52f415d50cf9b586f82da0ee2cd3bc5937", + "published": "2020-02-05T11:20:56.408153Z" + }, + { + "version": "0.2.1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.2.1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.2.1.tar.gz", + "archive_sha256": "8c673ab6837682efa92db82be185ae63743b7c85fad9af7cb389c68dc7209b80", + "published": "2020-02-10T15:34:33.966841Z" + }, + { + "version": "0.2.1+3", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.2.1+3", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.2.1%2B3.tar.gz", + "archive_sha256": "71bf59872482382826369560337ac40af099b3557ef1f2fe608e5b76aba0ba08", + "published": "2020-02-24T04:04:39.484669Z" + }, + { + "version": "0.2.1+hotfix.1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.2.1+hotfix.1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.2.1%2Bhotfix.1.tar.gz", + "archive_sha256": "27f896b81f04af7deb8605fdcc58f924805313f8ac08c47f446deb0d615052f1", + "published": "2020-02-12T13:16:52.629036Z" + }, + { + "version": "0.2.1+hotfix.2", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.2.1+hotfix.2", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.2.1%2Bhotfix.2.tar.gz", + "archive_sha256": "b79bf0ee5cac7d4414f0e7fe18d93467a2c0a815130a69671a8a73d1a4e97370", + "published": "2020-02-13T03:50:06.570996Z" + }, + { + "version": "0.3.1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.3.1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.3.1.tar.gz", + "archive_sha256": "0dad69cebcb80b920ae7ed3c6b479b030aa4c7cdf84aaf3eddc4e09be70e4f61", + "published": "2020-02-24T04:08:56.844608Z" + }, + { + "version": "0.3.1+1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.3.1+1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.3.1%2B1.tar.gz", + "archive_sha256": "03a1973a8b71741709d65370c5b1ece838bc62f56ea7038263cd8f0ba0d4f1c1", + "published": "2020-02-25T04:03:44.708601Z" + }, + { + "version": "0.5.1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.1.tar.gz", + "archive_sha256": "e2e9777c3712cb8aa9fc1b275ad8675b43a3e900adbcb8563624dea6b65ed877", + "published": "2020-03-09T09:17:02.774181Z" + }, + { + "version": "0.5.1+1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.1+1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.1%2B1.tar.gz", + "archive_sha256": "36ce31f26f2ccd39eb1650fc1f29879bad43439ec09b59a9fd3044843ce1d6c7", + "published": "2020-03-10T07:33:54.509246Z" + }, + { + "version": "0.5.1+2", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.1+2", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.1%2B2.tar.gz", + "archive_sha256": "577ac4b284c27761f96b1fc9e47c057bc34076cd84a9ffd41cdbe037917b9a8e", + "published": "2020-03-10T12:28:55.258668Z" + }, + { + "version": "0.5.1+3", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.1+3", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.1%2B3.tar.gz", + "archive_sha256": "75897f8e2e701b1fa9ad5f29846f2551450e3363981f8f03241cb5279c32b9e8", + "published": "2020-03-11T06:24:41.326411Z" + }, + { + "version": "0.5.1+4", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.1+4", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.1%2B4.tar.gz", + "archive_sha256": "b1505499de4e23e8fd9dfc03e01d98effcb330f28452c0a8d2698c159681ab3f", + "published": "2020-03-11T06:37:37.016887Z" + }, + { + "version": "0.5.1+5", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.1+5", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.1%2B5.tar.gz", + "archive_sha256": "5b1e83f1f0137b34b4a018be1282fde886beb55ba46d0b8b42f403b4ab1de551", + "published": "2020-03-12T06:24:21.642494Z" + }, + { + "version": "0.5.2", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.2", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.2.tar.gz", + "archive_sha256": "cf5e1165eae2959c016abd43aca5fbef8ddc415d5e7760e7a79ae2d2058861b3", + "published": "2020-03-13T06:23:18.744917Z" + }, + { + "version": "0.5.2+1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.2+1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.2%2B1.tar.gz", + "archive_sha256": "d6faa6e6b8b10a1fd64d1ce625fa8bdf796b0d68872eeed39b476de70f6575e4", + "published": "2020-03-17T08:16:05.352849Z" + }, + { + "version": "0.5.3", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.3", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.3.tar.gz", + "archive_sha256": "f90b2ca7758bb4762da63fb44f50a749860ae650cf46a0da1f2c8a8bf64b3534", + "published": "2020-03-19T04:00:25.045092Z" + }, + { + "version": "0.5.3+1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.3+1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.3%2B1.tar.gz", + "archive_sha256": "4591f420ac5c66a02e5aa72da2c4ac1dafc45c58c20a0301b658fcb4a193ac9f", + "published": "2020-03-24T04:01:42.936272Z" + }, + { + "version": "0.5.3+2", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.3+2", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.3%2B2.tar.gz", + "archive_sha256": "fe236c0aace46adea45b72cb2c255a8ce0252d7c482312851c673c8a5326ba08", + "published": "2020-03-31T07:51:37.161227Z" + }, + { + "version": "0.5.3+3", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.3+3", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.3%2B3.tar.gz", + "archive_sha256": "5470cdb704ec0b0db2ef24dc6abf52a20b22838812ffd72d5fce6bd7d2301ebf", + "published": "2020-04-25T11:00:45.370666Z" + }, + { + "version": "0.5.4", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.4", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.4.tar.gz", + "archive_sha256": "659a90232b4ef8bb5209998540c851220f8bc02909e942745dd1f9dc945fb50a", + "published": "2020-04-27T08:16:54.531384Z" + }, + { + "version": "0.5.4+1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.4+1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.4%2B1.tar.gz", + "archive_sha256": "2c13b30c4a0849e4679fa3674e1cf4f37616c7317c80097b6277ac82a1d77449", + "published": "2020-04-28T08:56:53.471649Z" + }, + { + "version": "0.5.5", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.5", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0", + "flutter": ">=1.12.0 <2.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.5.tar.gz", + "archive_sha256": "6afe114c088b8836cfda95e919ca7a27b194ab38a15652871cbe4a18a2614828", + "published": "2020-06-02T17:27:44.185641Z" + }, + { + "version": "0.5.5+1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.5+1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0", + "flutter": ">=1.12.0 <2.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.5%2B1.tar.gz", + "archive_sha256": "fcd4c89bac8912c0a738d20e616d0c0c7d2660ed66c78d61d6d70707e11d903b", + "published": "2020-06-03T12:55:34.627592Z" + }, + { + "version": "0.5.5+2", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.5+2", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0", + "flutter": ">=1.12.0 <2.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.5%2B2.tar.gz", + "archive_sha256": "0d8cc98dfea58b6e8c6d1d8e368ce4b78641fc1b0c67c7f6b66be71a2192131b", + "published": "2020-06-07T03:01:44.581962Z" + }, + { + "version": "0.5.5+3", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.5+3", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0", + "flutter": ">=1.12.0 <2.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.5%2B3.tar.gz", + "archive_sha256": "47ae200b33f76c716c3f3cc5b9a01edbda01d4229a3e7809e83eee0ca9d3d67a", + "published": "2020-06-11T12:10:22.714748Z" + }, + { + "version": "0.5.6", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.6", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0", + "flutter": ">=1.12.0 <2.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.6.tar.gz", + "archive_sha256": "999566d6723a81a6790f9211dd88a86e50ac11f576e348961db98d1d312046d9", + "published": "2020-10-22T09:34:01.825061Z" + }, + { + "version": "0.5.7", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.7", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0", + "flutter": ">=1.12.0 <2.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.7.tar.gz", + "archive_sha256": "a7a4bc134d9b2654c22cbde25e9fbc5ef810940d8329072bb6eb3dc0cdb39363", + "published": "2020-10-22T14:05:29.407637Z" + }, + { + "version": "0.5.7+1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.5.7+1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0", + "flutter": ">=1.12.0 <2.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.5.7%2B1.tar.gz", + "archive_sha256": "6cc3a0d629842cd6a69c6a59a90fb9b34cec377ff9ce08960dbe6e5bec5676f5", + "published": "2020-10-23T09:58:57.828463Z" + }, + { + "version": "0.6.1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.6.1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0", + "flutter": ">=1.12.0 <2.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + }, + "flutter_web_plugins": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + }, + "web": { + "pluginClass": "AudioManagerPlugin", + "fileName": "audio_manager_web.dart" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.6.1.tar.gz", + "archive_sha256": "3327db1c1f065bcb434f240899d65dca3590dd28e1ab9fbdb82c79b47238055a", + "published": "2020-10-27T08:30:31.727200Z" + }, + { + "version": "0.6.2", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.6.2", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0", + "flutter": ">=1.12.0 <2.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + }, + "flutter_web_plugins": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + }, + "web": { + "pluginClass": "AudioManagerPlugin", + "fileName": "audio_manager_web.dart" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.6.2.tar.gz", + "archive_sha256": "84240bf9f5b048c696b3d0bc2063d2f6abb60de0a7c23cf6cb8abfe299233dee", + "published": "2020-11-26T09:58:23.851983Z" + }, + { + "version": "0.7.1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.7.1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0", + "flutter": ">=1.12.0 <2.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + }, + "flutter_web_plugins": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + }, + "web": { + "pluginClass": "AudioManagerPlugin", + "fileName": "audio_manager_web.dart" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.7.1.tar.gz", + "archive_sha256": "e12d6734fbe19af96f9916f31b282da8b446d1a77c75326503271730adfd7788", + "published": "2021-01-26T09:10:40.222865Z" + }, + { + "version": "0.7.2", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.7.2", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0", + "flutter": ">=1.12.0 <2.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + }, + "flutter_web_plugins": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + }, + "web": { + "pluginClass": "AudioManagerPlugin", + "fileName": "audio_manager_web.dart" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.7.2.tar.gz", + "archive_sha256": "b343e13e074008531d03502d559543aaa0cb11ec7050431ccb2e3e5e097ff618", + "published": "2021-01-27T03:50:37.801441Z" + }, + { + "version": "0.7.3", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.7.3", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.1.0 <3.0.0", + "flutter": ">=1.12.0 <2.0.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + }, + "flutter_web_plugins": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + }, + "web": { + "pluginClass": "AudioManagerPlugin", + "fileName": "audio_manager_web.dart" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.7.3.tar.gz", + "archive_sha256": "8cfaf06e8ff49fa57ab2ef87e05ab813e50ceccffea15efee2c6b71b2c63fe18", + "published": "2021-01-27T03:57:26.848937Z" + }, + { + "version": "0.8.1", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.8.1", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.12.0 <3.0.0", + "flutter": ">=1.20.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + }, + "flutter_web_plugins": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + }, + "web": { + "pluginClass": "AudioManagerPlugin", + "fileName": "audio_manager_web.dart" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.8.1.tar.gz", + "archive_sha256": "22deaa1d72b07f84eb54fc8cdf643786ede522d3ef76d70ccb8edd3469484e7c", + "published": "2021-05-27T09:13:54.510298Z" + }, + { + "version": "0.8.2", + "pubspec": { + "name": "audio_manager", + "description": "A flutter plugin for music playback, including notification handling.", + "version": "0.8.2", + "homepage": "https://github.com/jeromexiong/audio_manager", + "environment": { + "sdk": ">=2.12.0 <3.0.0", + "flutter": ">=1.20.0" + }, + "dependencies": { + "flutter": { + "sdk": "flutter" + }, + "flutter_web_plugins": { + "sdk": "flutter" + } + }, + "dev_dependencies": { + "flutter_test": { + "sdk": "flutter" + } + }, + "flutter": { + "plugin": { + "platforms": { + "android": { + "package": "cc.dync.audio_manager", + "pluginClass": "AudioManagerPlugin" + }, + "ios": { + "pluginClass": "AudioManagerPlugin" + }, + "web": { + "pluginClass": "AudioManagerPlugin", + "fileName": "audio_manager_web.dart" + } + } + } + } + }, + "archive_url": "https://pub.dartlang.org/packages/audio_manager/versions/0.8.2.tar.gz", + "archive_sha256": "6f26b61cc39e258619d0e1bc34500bb7f3926dcc4b6a8c7ea31b67c62381c931", + "published": "2021-10-05T11:25:10.278136Z" + } + ] +} diff --git a/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_authentication b/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_authentication index b4b312a..25ca01a 100644 --- a/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_authentication +++ b/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_authentication @@ -1,77 +1,77 @@ { "name": "authentication", "latest": { "version": "0.0.1", "pubspec": { "name": "authentication", "description": "Persistent user authentication for Flutter with optional backend API integration.", "version": "0.0.1", "author": null, "homepage": null, "environment": { "sdk": ">=2.7.0 <3.0.0", "flutter": ">=1.17.0 <2.0.0" }, "dependencies": { "flutter": { "sdk": "flutter" } }, "dev_dependencies": { "flutter_test": { "sdk": "flutter" } }, "flutter": { "plugin": { "platforms": { "some_platform": { "pluginClass": "somePluginClass" } } } } }, "archive_url": "https://pub.dartlang.org/packages/authentication/versions/0.0.1.tar.gz", "archive_sha256": "0179334b346cb67e4e6e3c905e5cc5c8e488a45ebd99fd2be3a7e0476d620d99", "published": "2020-08-13T04:53:34.134687Z" }, "versions": [ { "version": "0.0.1", "pubspec": { "name": "authentication", "description": "Persistent user authentication for Flutter with optional backend API integration.", "version": "0.0.1", "author": null, "homepage": null, "environment": { "sdk": ">=2.7.0 <3.0.0", "flutter": ">=1.17.0 <2.0.0" }, "dependencies": { "flutter": { "sdk": "flutter" } }, "dev_dependencies": { "flutter_test": { "sdk": "flutter" } }, "flutter": { "plugin": { "platforms": { "some_platform": { "pluginClass": "somePluginClass" } } } } }, "archive_url": "https://pub.dartlang.org/packages/authentication/versions/0.0.1.tar.gz", "archive_sha256": "0179334b346cb67e4e6e3c905e5cc5c8e488a45ebd99fd2be3a7e0476d620d99", "published": "2020-08-13T04:53:34.134687Z" } ] -} \ No newline at end of file +} diff --git a/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_bezier b/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_bezier index efd84a4..dacdd55 100644 --- a/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_bezier +++ b/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_bezier @@ -1,55 +1,55 @@ { "name": "bezier", "latest": { "version": "1.1.5", "pubspec": { "name": "bezier", "version": "1.1.5", "authors": [ "Aaron Barrett ", "Isaac Barrett " ], - "description": "A 2D Bézier curve math library. Based heavily on the work of @TheRealPomax .\nLive examples can be found at .", + "description": "A 2D Bézier curve math library. Based heavily on the work of @TheRealPomax . Live examples can be found at .", "homepage": "https://github.com/aab29/bezier.dart", "environment": { "sdk": ">=2.0.0 <3.0.0" }, "dependencies": { "vector_math": "^2.0.0" }, "dev_dependencies": { "test": "^1.0.0" } }, "archive_url": "https://pub.dartlang.org/packages/bezier/versions/1.1.5.tar.gz", "archive_sha256": "cc5da2fa927b5d347550f78d456cd984b7df78a7f0405119cdab12111e2f9ee8", "published": "2019-12-22T03:17:30.805225Z" }, "versions": [ { "version": "1.1.5", "pubspec": { "name": "bezier", "version": "1.1.5", "authors": [ "Aaron Barrett ", "Isaac Barrett " ], - "description": "A 2D Bézier curve math library. Based heavily on the work of @TheRealPomax .\nLive examples can be found at .", + "description": "A 2D Bézier curve math library. Based heavily on the work of @TheRealPomax .Live examples can be found at .", "homepage": "https://github.com/aab29/bezier.dart", "environment": { "sdk": ">=2.0.0 <3.0.0" }, "dependencies": { "vector_math": "^2.0.0" }, "dev_dependencies": { "test": "^1.0.0" } }, "archive_url": "https://pub.dartlang.org/packages/bezier/versions/1.1.5.tar.gz", "archive_sha256": "cc5da2fa927b5d347550f78d456cd984b7df78a7f0405119cdab12111e2f9ee8", "published": "2019-12-22T03:17:30.805225Z" } ] -} \ No newline at end of file +} diff --git a/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_pdf b/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_pdf index c015890..1541536 100644 --- a/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_pdf +++ b/swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_pdf @@ -1,88 +1,88 @@ { "name": "pdf", "latest": { "version": "3.8.2", "pubspec": { "name": "pdf", "description": "A pdf producer for Dart. It can create pdf files for both web or flutter.", "homepage": "https://github.com/DavBfr/dart_pdf/tree/master/pdf", "repository": "https://github.com/DavBfr/dart_pdf", "issue_tracker": "https://github.com/DavBfr/dart_pdf/issues", "version": "3.8.2", "environment": { "sdk": ">=2.12.0 <3.0.0" }, "dependencies": { "archive": "^3.1.0", "barcode": ">=2.2.0 <3.0.0", "crypto": "^3.0.0", "image": ">=3.0.1 <4.0.0", "meta": ">=1.3.0 <2.0.0", "path_parsing": ">=0.2.0 <2.0.0", "vector_math": "^2.1.0", "xml": ">=5.1.0 <7.0.0" }, "dev_dependencies": { "flutter_lints": "^1.0.4", "test": ">=1.16.0 <2.0.0" } }, "archive_url": "https://pub.dartlang.org/packages/pdf/versions/3.8.2.tar.gz", "published": "2022-07-25T11:38:25.983876Z" }, "versions": [ { "version": "1.0.0", "pubspec": { "version": "1.0.0", "name": "pdf", "dependencies": { "ttf_parser": "^1.0.0", "vector_math": "^2.0.7", "meta": "^1.1.5" }, "author": "David PHAM-VAN ", "description": "A pdf producer for Dart", "homepage": "https://github.com/davbfr/dart_pdf", "environment": { "sdk": ">=1.8.0 <2.0.0" }, "dev_dependencies": { "test": "any" } }, "archive_url": "https://pub.dartlang.org/packages/pdf/versions/1.0.0.tar.gz", "published": "2018-07-16T21:12:28.894137Z" }, { "version": "3.8.2", "pubspec": { "name": "pdf", "description": "A pdf producer for Dart. It can create pdf files for both web or flutter.", "homepage": "https://github.com/DavBfr/dart_pdf/tree/master/pdf", "repository": "https://github.com/DavBfr/dart_pdf", "issue_tracker": "https://github.com/DavBfr/dart_pdf/issues", "version": "3.8.2", "environment": { "sdk": ">=2.12.0 <3.0.0" }, "dependencies": { "archive": "^3.1.0", "barcode": ">=2.2.0 <3.0.0", "crypto": "^3.0.0", "image": ">=3.0.1 <4.0.0", "meta": ">=1.3.0 <2.0.0", "path_parsing": ">=0.2.0 <2.0.0", "vector_math": "^2.1.0", "xml": ">=5.1.0 <7.0.0" }, "dev_dependencies": { "flutter_lints": "^1.0.4", "test": ">=1.16.0 <2.0.0" } }, "archive_url": "https://pub.dartlang.org/packages/pdf/versions/3.8.2.tar.gz", "published": "2022-07-25T11:38:25.983876Z" } ] -} \ No newline at end of file +} diff --git a/swh/loader/package/pubdev/tests/test_pubdev.py b/swh/loader/package/pubdev/tests/test_pubdev.py index 0979dfd..9267c24 100644 --- a/swh/loader/package/pubdev/tests/test_pubdev.py +++ b/swh/loader/package/pubdev/tests/test_pubdev.py @@ -1,272 +1,326 @@ # Copyright (C) 2022 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU General Public License version 3, or any later version # See top-level LICENSE file for more information import pytest from swh.loader.package.pubdev.loader import PubDevLoader from swh.loader.package.utils import EMPTY_AUTHOR from swh.loader.tests import assert_last_visit_matches, check_snapshot, get_stats from swh.model.hashutil import hash_to_bytes from swh.model.model import ( ObjectType, Person, Release, Snapshot, SnapshotBranch, TargetType, TimestampWithTimezone, ) EXPECTED_PACKAGES = [ { "url": "https://pub.dev/packages/Autolinker", # one version }, { "url": "https://pub.dev/packages/pdf", # multiple versions }, { "url": "https://pub.dev/packages/bezier", # multiple authors }, { "url": "https://pub.dev/packages/authentication", # empty author }, { "url": "https://pub.dev/packages/abstract_io", # loose versions names }, + { + "url": "https://pub.dev/packages/audio_manager", # loose ++ versions names + }, ] def test_get_versions(requests_mock_datadir, swh_storage): loader = PubDevLoader( swh_storage, url=EXPECTED_PACKAGES[1]["url"], ) assert loader.get_versions() == [ "1.0.0", "3.8.2", ] -def test_get_loose_versions(requests_mock_datadir, swh_storage): +def test_sort_loose_versions(requests_mock_datadir, swh_storage): """Sometimes version name does not follow semver""" loader = PubDevLoader( swh_storage, url=EXPECTED_PACKAGES[4]["url"], ) assert loader.get_versions() == ["0.1.2+4", "0.1.2+5", "0.1.2+6"] +def test_sort_loose_versions_1(requests_mock_datadir, swh_storage): + """Sometimes version name does not follow semver and mix patterns""" + loader = PubDevLoader( + swh_storage, + url=EXPECTED_PACKAGES[5]["url"], + ) + assert loader.get_versions() == [ + "0.0.1", + "0.0.2", + "0.1.1", + "0.1.2", + "0.1.3", + "0.1.4", + "0.1.5", + "0.2.1", + "0.2.1+hotfix.1", + "0.2.1+hotfix.2", + "0.2.1+3", + "0.3.1", + "0.3.1+1", + "0.5.1", + "0.5.1+1", + "0.5.1+2", + "0.5.1+3", + "0.5.1+4", + "0.5.1+5", + "0.5.2", + "0.5.2+1", + "0.5.3", + "0.5.3+1", + "0.5.3+2", + "0.5.3+3", + "0.5.4", + "0.5.4+1", + "0.5.5", + "0.5.5+1", + "0.5.5+2", + "0.5.5+3", + "0.5.6", + "0.5.7", + "0.5.7+1", + "0.6.1", + "0.6.2", + "0.7.1", + "0.7.2", + "0.7.3", + "0.8.1", + "0.8.2", + ] + + def test_get_default_version(requests_mock_datadir, swh_storage): loader = PubDevLoader( swh_storage, url=EXPECTED_PACKAGES[1]["url"], ) assert loader.get_default_version() == "3.8.2" def test_pubdev_loader_load_one_version(datadir, requests_mock_datadir, swh_storage): loader = PubDevLoader( swh_storage, url=EXPECTED_PACKAGES[0]["url"], ) load_status = loader.load() assert load_status["status"] == "eventful" assert load_status["snapshot_id"] is not None expected_snapshot_id = "245092931ba809e6c54ebda8f865fb5a969a4134" expected_release_id = "919f267ea050539606344d49d14bf594c4386e5a" assert expected_snapshot_id == load_status["snapshot_id"] expected_snapshot = Snapshot( id=hash_to_bytes(load_status["snapshot_id"]), branches={ b"releases/0.1.1": SnapshotBranch( target=hash_to_bytes(expected_release_id), target_type=TargetType.RELEASE, ), b"HEAD": SnapshotBranch( target=b"releases/0.1.1", target_type=TargetType.ALIAS, ), }, ) check_snapshot(expected_snapshot, swh_storage) stats = get_stats(swh_storage) assert { "content": 1, "directory": 1, "origin": 1, "origin_visit": 1, "release": 1, "revision": 0, "skipped_content": 0, "snapshot": 1, } == stats assert swh_storage.release_get([hash_to_bytes(expected_release_id)])[0] == Release( name=b"0.1.1", message=b"Synthetic release for pub.dev source package Autolinker version" b" 0.1.1\n\nPort of Autolinker.js to dart\n", target=hash_to_bytes("3fb6d4f2c0334d1604357ae92b2dd38a55a78194"), target_type=ObjectType.DIRECTORY, synthetic=True, author=Person( fullname=b"hackcave ", name=b"hackcave", email=b"hackers@hackcave.org", ), date=TimestampWithTimezone.from_iso8601("2014-12-24T22:34:02.534090+00:00"), id=hash_to_bytes(expected_release_id), ) assert_last_visit_matches( swh_storage, url=EXPECTED_PACKAGES[0]["url"], status="full", type="pubdev", snapshot=expected_snapshot.id, ) def test_pubdev_loader_load_multiple_versions( datadir, requests_mock_datadir, swh_storage ): loader = PubDevLoader( swh_storage, url=EXPECTED_PACKAGES[1]["url"], ) load_status = loader.load() assert load_status["status"] == "eventful" assert load_status["snapshot_id"] is not None expected_snapshot_id = "43d5b68a9fa973aa95e56916aaef70841ccbc2a0" assert expected_snapshot_id == load_status["snapshot_id"] expected_snapshot = Snapshot( id=hash_to_bytes(load_status["snapshot_id"]), branches={ b"releases/1.0.0": SnapshotBranch( target=hash_to_bytes("fbf8e40af675096681954553d737861e10b57216"), target_type=TargetType.RELEASE, ), b"releases/3.8.2": SnapshotBranch( target=hash_to_bytes("627a5d586e3fb4e7319b17f1aee268fe2fb8e01c"), target_type=TargetType.RELEASE, ), b"HEAD": SnapshotBranch( target=b"releases/3.8.2", target_type=TargetType.ALIAS, ), }, ) check_snapshot(expected_snapshot, swh_storage) stats = get_stats(swh_storage) assert { "content": 1 + 1, "directory": 1 + 1, "origin": 1, "origin_visit": 1, "release": 1 + 1, "revision": 0, "skipped_content": 0, "snapshot": 1, } == stats assert_last_visit_matches( swh_storage, url=EXPECTED_PACKAGES[1]["url"], status="full", type="pubdev", snapshot=expected_snapshot.id, ) def test_pubdev_loader_multiple_authors(datadir, requests_mock_datadir, swh_storage): loader = PubDevLoader( swh_storage, url=EXPECTED_PACKAGES[2]["url"], ) load_status = loader.load() assert load_status["status"] == "eventful" assert load_status["snapshot_id"] is not None expected_snapshot_id = "4fa9f19d1d6ccc70921c8c50b278f510db63aa36" expected_release_id = "538c98fd69a42d8d0561a7ca95b354de2143a3ab" assert expected_snapshot_id == load_status["snapshot_id"] expected_snapshot = Snapshot( id=hash_to_bytes(load_status["snapshot_id"]), branches={ b"releases/1.1.5": SnapshotBranch( target=hash_to_bytes(expected_release_id), target_type=TargetType.RELEASE, ), b"HEAD": SnapshotBranch( target=b"releases/1.1.5", target_type=TargetType.ALIAS, ), }, ) check_snapshot(expected_snapshot, swh_storage) release = swh_storage.release_get([hash_to_bytes(expected_release_id)])[0] assert release.author == Person( fullname=b"Aaron Barrett ", name=b"Aaron Barrett", email=b"aaron@aaronbarrett.com", ) def test_pubdev_loader_empty_author(datadir, requests_mock_datadir, swh_storage): loader = PubDevLoader( swh_storage, url=EXPECTED_PACKAGES[3]["url"], ) load_status = loader.load() assert load_status["status"] == "eventful" assert load_status["snapshot_id"] is not None expected_snapshot_id = "0c7fa6b9fced23c648d2093ad5597622683f8aed" expected_release_id = "7d8c05181069aa1049a3f0bc1d13bedc34625d47" assert expected_snapshot_id == load_status["snapshot_id"] expected_snapshot = Snapshot( id=hash_to_bytes(load_status["snapshot_id"]), branches={ b"releases/0.0.1": SnapshotBranch( target=hash_to_bytes(expected_release_id), target_type=TargetType.RELEASE, ), b"HEAD": SnapshotBranch( target=b"releases/0.0.1", target_type=TargetType.ALIAS, ), }, ) check_snapshot(expected_snapshot, swh_storage) release = swh_storage.release_get([hash_to_bytes(expected_release_id)])[0] assert release.author == EMPTY_AUTHOR def test_pubdev_invalid_origin(swh_storage): with pytest.raises(AssertionError): PubDevLoader( swh_storage, "http://nowhere/api/packages/42", ) diff --git a/swh/loader/package/tests/test_loader.py b/swh/loader/package/tests/test_loader.py index ccf47c3..1ea43dc 100644 --- a/swh/loader/package/tests/test_loader.py +++ b/swh/loader/package/tests/test_loader.py @@ -1,520 +1,541 @@ # Copyright (C) 2019-2021 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU General Public License version 3, or any later version # See top-level LICENSE file for more information import datetime import hashlib import logging import string from unittest.mock import Mock, call, patch import attr import pytest +from swh.loader.core.loader import ( + SENTRY_ORIGIN_URL_TAG_NAME, + SENTRY_VISIT_TYPE_TAG_NAME, +) from swh.loader.package.loader import BasePackageInfo, PackageLoader from swh.model.model import ( Origin, OriginVisit, OriginVisitStatus, Person, Release, Revision, RevisionType, Snapshot, SnapshotBranch, TargetType, TimestampWithTimezone, ) from swh.model.model import ExtID from swh.model.model import ObjectType as ModelObjectType from swh.model.swhids import CoreSWHID, ObjectType from swh.storage import get_storage from swh.storage.algos.snapshot import snapshot_get_latest class FakeStorage: def origin_add(self, origins): raise ValueError("We refuse to add an origin") def origin_visit_get_latest(self, origin): return None class FakeStorage2(FakeStorage): def origin_add(self, origins): pass def origin_visit_add(self, visits): raise ValueError("We refuse to add an origin visit") class StubPackageInfo(BasePackageInfo): pass class StubPackageLoader(PackageLoader[StubPackageInfo]): visit_type = "stub" def get_versions(self): return ["v1.0", "v2.0", "v3.0", "v4.0"] def get_package_info(self, version): p_info = StubPackageInfo( "http://example.org", f"example-{version}.tar", version=version ) extid_type = "extid-type1" if version in ("v1.0", "v2.0") else "extid-type2" # Versions 1.0 and 2.0 have an extid of a given type, v3.0 has an extid # of a different type patch.object( p_info, "extid", return_value=(extid_type, 0, f"extid-of-{version}".encode()), autospec=True, ).start() yield (f"branch-{version}", p_info) def _load_release(self, p_info, origin): return None def test_loader_origin_visit_failure(swh_storage): """Failure to add origin or origin visit should failed immediately""" loader = StubPackageLoader(swh_storage, "some-url") loader.storage = FakeStorage() actual_load_status = loader.load() assert actual_load_status == {"status": "failed"} loader.storage = FakeStorage2() actual_load_status2 = loader.load() assert actual_load_status2 == {"status": "failed"} def test_resolve_object_from_extids() -> None: storage = get_storage("memory") target = b"\x01" * 20 rel1 = Release( name=b"aaaa", message=b"aaaa", target=target, target_type=ModelObjectType.DIRECTORY, synthetic=False, ) rel2 = Release( name=b"bbbb", message=b"bbbb", target=target, target_type=ModelObjectType.DIRECTORY, synthetic=False, ) storage.release_add([rel1, rel2]) loader = StubPackageLoader(storage, "http://example.org/") p_info = Mock(wraps=BasePackageInfo(None, None, None)) # type: ignore # The PackageInfo does not support extids p_info.extid.return_value = None known_extids = {("extid-type", 0, b"extid-of-aaaa"): [rel1.swhid()]} whitelist = {b"unused"} assert loader.resolve_object_from_extids(known_extids, p_info, whitelist) is None # Some known extid, and the PackageInfo is not one of them (ie. cache miss) p_info.extid.return_value = ("extid-type", 0, b"extid-of-cccc") assert loader.resolve_object_from_extids(known_extids, p_info, whitelist) is None # Some known extid, and the PackageInfo is one of them (ie. cache hit), # but the target release was not in the previous snapshot p_info.extid.return_value = ("extid-type", 0, b"extid-of-aaaa") assert loader.resolve_object_from_extids(known_extids, p_info, whitelist) is None # Some known extid, and the PackageInfo is one of them (ie. cache hit), # and the target release was in the previous snapshot whitelist = {rel1.id} assert ( loader.resolve_object_from_extids(known_extids, p_info, whitelist) == rel1.swhid() ) # Same as before, but there is more than one extid, and only one is an allowed # release whitelist = {rel1.id} known_extids = {("extid-type", 0, b"extid-of-aaaa"): [rel2.swhid(), rel1.swhid()]} assert ( loader.resolve_object_from_extids(known_extids, p_info, whitelist) == rel1.swhid() ) def test_resolve_object_from_extids_missing_target() -> None: storage = get_storage("memory") target = b"\x01" * 20 rel = Release( name=b"aaaa", message=b"aaaa", target=target, target_type=ModelObjectType.DIRECTORY, synthetic=False, ) loader = StubPackageLoader(storage, "http://example.org/") p_info = Mock(wraps=BasePackageInfo(None, None, None)) # type: ignore known_extids = {("extid-type", 0, b"extid-of-aaaa"): [rel.swhid()]} p_info.extid.return_value = ("extid-type", 0, b"extid-of-aaaa") whitelist = {rel.id} # Targeted release is missing from the storage assert loader.resolve_object_from_extids(known_extids, p_info, whitelist) is None storage.release_add([rel]) # Targeted release now exists assert ( loader.resolve_object_from_extids(known_extids, p_info, whitelist) == rel.swhid() ) def test_load_get_known_extids() -> None: """Checks PackageLoader.load() fetches known extids efficiently""" storage = Mock(wraps=get_storage("memory")) loader = StubPackageLoader(storage, "http://example.org") loader.load() # Calls should be grouped by extid type storage.extid_get_from_extid.assert_has_calls( [ call("extid-type1", [b"extid-of-v1.0", b"extid-of-v2.0"], version=0), call("extid-type2", [b"extid-of-v3.0", b"extid-of-v4.0"], version=0), ], any_order=True, ) def test_load_extids() -> None: """Checks PackageLoader.load() skips iff it should, and writes (only) the new ExtIDs""" storage = get_storage("memory") dir_swhid = CoreSWHID(object_type=ObjectType.DIRECTORY, object_id=b"e" * 20) rels = [ Release( name=f"v{i}.0".encode(), message=b"blah\n", target=dir_swhid.object_id, target_type=ModelObjectType.DIRECTORY, synthetic=True, ) for i in (1, 2, 3, 4) ] storage.release_add(rels[0:3]) origin = "http://example.org" rel1_swhid = rels[0].swhid() rel2_swhid = rels[1].swhid() rel3_swhid = rels[2].swhid() rel4_swhid = rels[3].swhid() # Results of a previous load storage.extid_add( [ ExtID("extid-type1", b"extid-of-v1.0", rel1_swhid), ExtID("extid-type2", b"extid-of-v2.0", rel2_swhid), ] ) last_snapshot = Snapshot( branches={ b"v1.0": SnapshotBranch( target_type=TargetType.RELEASE, target=rel1_swhid.object_id ), b"v2.0": SnapshotBranch( target_type=TargetType.RELEASE, target=rel2_swhid.object_id ), b"v3.0": SnapshotBranch( target_type=TargetType.RELEASE, target=rel3_swhid.object_id ), } ) storage.snapshot_add([last_snapshot]) date = datetime.datetime.now(tz=datetime.timezone.utc) storage.origin_add([Origin(url=origin)]) storage.origin_visit_add( [OriginVisit(origin="http://example.org", visit=1, date=date, type="tar")] ) storage.origin_visit_status_add( [ OriginVisitStatus( origin=origin, visit=1, status="full", date=date, snapshot=last_snapshot.id, ) ] ) loader = StubPackageLoader(storage, "http://example.org") patch.object( loader, "_load_release", return_value=(rel4_swhid.object_id, dir_swhid.object_id), autospec=True, ).start() loader.load() assert loader._load_release.mock_calls == [ # type: ignore # v1.0: not loaded because there is already its (extid_type, extid, rel) # in the storage. # v2.0: loaded, because there is already a similar extid, but different type call( StubPackageInfo(origin, "example-v2.0.tar", "v2.0"), Origin(url=origin), ), # v3.0: loaded despite having an (extid_type, extid) in storage, because # the target of the extid is not in the previous snapshot call( StubPackageInfo(origin, "example-v3.0.tar", "v3.0"), Origin(url=origin), ), # v4.0: loaded, because there isn't its extid call( StubPackageInfo(origin, "example-v4.0.tar", "v4.0"), Origin(url=origin), ), ] # then check the snapshot has all the branches. # versions 2.0 to 4.0 all point to rel4_swhid (instead of the value of the last # snapshot), because they had to be loaded (mismatched extid), and the mocked # _load_release always returns rel4_swhid. snapshot = Snapshot( branches={ b"branch-v1.0": SnapshotBranch( target_type=TargetType.RELEASE, target=rel1_swhid.object_id ), b"branch-v2.0": SnapshotBranch( target_type=TargetType.RELEASE, target=rel4_swhid.object_id ), b"branch-v3.0": SnapshotBranch( target_type=TargetType.RELEASE, target=rel4_swhid.object_id ), b"branch-v4.0": SnapshotBranch( target_type=TargetType.RELEASE, target=rel4_swhid.object_id ), } ) assert snapshot_get_latest(storage, origin) == snapshot extids = storage.extid_get_from_target( ObjectType.RELEASE, [ rel1_swhid.object_id, rel2_swhid.object_id, rel3_swhid.object_id, rel4_swhid.object_id, ], ) assert set(extids) == { # What we inserted at the beginning of the test: ExtID("extid-type1", b"extid-of-v1.0", rel1_swhid), ExtID("extid-type2", b"extid-of-v2.0", rel2_swhid), # Added by the loader: ExtID("extid-type1", b"extid-of-v2.0", rel4_swhid), ExtID("extid-type2", b"extid-of-v3.0", rel4_swhid), ExtID("extid-type2", b"extid-of-v4.0", rel4_swhid), } def test_load_upgrade_from_revision_extids(caplog): """Tests that, when loading incrementally based on a snapshot made by an old version of the loader, the loader will convert revisions to releases and add them to the storage. Also checks that, if an extid exists pointing to a non-existent revision (which should never happen, but you never know...), the release is loaded from scratch.""" storage = get_storage("memory") origin = "http://example.org" dir1_swhid = CoreSWHID(object_type=ObjectType.DIRECTORY, object_id=b"d" * 20) dir2_swhid = CoreSWHID(object_type=ObjectType.DIRECTORY, object_id=b"e" * 20) date = TimestampWithTimezone.from_datetime( datetime.datetime.now(tz=datetime.timezone.utc) ) person = Person.from_fullname(b"Jane Doe ") rev1 = Revision( message=b"blah", author=person, date=date, committer=person, committer_date=date, directory=dir1_swhid.object_id, type=RevisionType.TAR, synthetic=True, ) rel1 = Release( name=b"v1.0", message=b"blah\n", author=person, date=date, target=dir1_swhid.object_id, target_type=ModelObjectType.DIRECTORY, synthetic=True, ) rev1_swhid = rev1.swhid() rel1_swhid = rel1.swhid() rev2_swhid = CoreSWHID(object_type=ObjectType.REVISION, object_id=b"b" * 20) rel2_swhid = CoreSWHID(object_type=ObjectType.RELEASE, object_id=b"c" * 20) # Results of a previous load storage.extid_add( [ ExtID("extid-type1", b"extid-of-v1.0", rev1_swhid, 0), ExtID("extid-type1", b"extid-of-v2.0", rev2_swhid, 0), ] ) storage.revision_add([rev1]) last_snapshot = Snapshot( branches={ b"v1.0": SnapshotBranch( target_type=TargetType.REVISION, target=rev1_swhid.object_id ), b"v2.0": SnapshotBranch( target_type=TargetType.REVISION, target=rev2_swhid.object_id ), } ) storage.snapshot_add([last_snapshot]) date = datetime.datetime.now(tz=datetime.timezone.utc) storage.origin_add([Origin(url=origin)]) storage.origin_visit_add( [OriginVisit(origin="http://example.org", visit=1, date=date, type="tar")] ) storage.origin_visit_status_add( [ OriginVisitStatus( origin=origin, visit=1, status="full", date=date, snapshot=last_snapshot.id, ) ] ) loader = StubPackageLoader(storage, "http://example.org") patch.object( loader, "_load_release", return_value=(rel2_swhid.object_id, dir2_swhid.object_id), autospec=True, ).start() patch.object( loader, "get_versions", return_value=["v1.0", "v2.0", "v3.0"], autospec=True, ).start() caplog.set_level(logging.ERROR) loader.load() assert len(caplog.records) == 1 (record,) = caplog.records assert record.levelname == "ERROR" assert "Failed to upgrade branch branch-v2.0" in record.message assert loader._load_release.mock_calls == [ # v1.0: not loaded because there is already a revision matching it # v2.0: loaded, as the revision is missing from the storage even though there # is an extid call(StubPackageInfo(origin, "example-v2.0.tar", "v2.0"), Origin(url=origin)), # v3.0: loaded (did not exist yet) call(StubPackageInfo(origin, "example-v3.0.tar", "v3.0"), Origin(url=origin)), ] snapshot = Snapshot( branches={ b"branch-v1.0": SnapshotBranch( target_type=TargetType.RELEASE, target=rel1_swhid.object_id ), b"branch-v2.0": SnapshotBranch( target_type=TargetType.RELEASE, target=rel2_swhid.object_id ), b"branch-v3.0": SnapshotBranch( target_type=TargetType.RELEASE, target=rel2_swhid.object_id ), } ) assert snapshot_get_latest(storage, origin) == snapshot extids = storage.extid_get_from_target( ObjectType.RELEASE, [ rel1_swhid.object_id, rel2_swhid.object_id, ], ) assert set(extids) == { ExtID("extid-type1", b"extid-of-v1.0", rel1_swhid), ExtID("extid-type1", b"extid-of-v2.0", rel2_swhid), ExtID("extid-type2", b"extid-of-v3.0", rel2_swhid), } def test_manifest_extid(): """Compute primary key should return the right identity""" @attr.s class TestPackageInfo(BasePackageInfo): a = attr.ib() b = attr.ib() length = attr.ib() filename = attr.ib() MANIFEST_FORMAT = string.Template("$a $b") p_info = TestPackageInfo( url="http://example.org/", a=1, b=2, length=221837, filename="8sync-0.1.0.tar.gz", version="0.1.0", ) actual_id = p_info.extid() assert actual_id == ("package-manifest-sha256", 0, hashlib.sha256(b"1 2").digest()) def test_no_env_swh_config_filename_raise(monkeypatch): """No SWH_CONFIG_FILENAME environment variable makes package loader init raise""" class DummyPackageLoader(PackageLoader): """A dummy package loader for test purpose""" pass monkeypatch.delenv("SWH_CONFIG_FILENAME", raising=False) with pytest.raises( AssertionError, match="SWH_CONFIG_FILENAME environment variable is undefined" ): DummyPackageLoader.from_configfile(url="some-url") + + +class StubPackageLoaderWithError(StubPackageLoader): + def get_versions(self, *args, **kwargs): + raise Exception("error") + + +def test_loader_sentry_tags_on_error(swh_storage, sentry_events): + origin_url = "http://example.org/package/name" + loader = StubPackageLoaderWithError(swh_storage, origin_url) + loader.load() + sentry_tags = sentry_events[0]["tags"] + assert sentry_tags.get(SENTRY_ORIGIN_URL_TAG_NAME) == origin_url + assert ( + sentry_tags.get(SENTRY_VISIT_TYPE_TAG_NAME) + == StubPackageLoaderWithError.visit_type + )