diff --git a/PKG-INFO b/PKG-INFO index e14129d..92d127e 100644 --- a/PKG-INFO +++ b/PKG-INFO @@ -1,42 +1,42 @@ Metadata-Version: 2.1 Name: swh.model -Version: 6.5.0 +Version: 6.5.1 Summary: Software Heritage data model Home-page: https://forge.softwareheritage.org/diffusion/DMOD/ Author: Software Heritage developers Author-email: swh-devel@inria.fr Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest Project-URL: Funding, https://www.softwareheritage.org/donate Project-URL: Source, https://forge.softwareheritage.org/source/swh-model Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-model/ Classifier: Programming Language :: Python :: 3 Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Classifier: Operating System :: OS Independent Classifier: Development Status :: 5 - Production/Stable Requires-Python: >=3.7 Description-Content-Type: text/markdown Provides-Extra: cli Provides-Extra: testing-minimal Provides-Extra: testing License-File: LICENSE License-File: AUTHORS swh-model ========= Implementation of the Data model of the Software Heritage project, used to archive source code artifacts. This module defines the notion of SoftWare Heritage persistent IDentifiers (SWHIDs) and provides tools to compute them: ```sh $ swh-identify fork.c kmod.c sched/deadline.c swh:1:cnt:2e391c754ae730bd2d8520c2ab497c403220c6e3 fork.c swh:1:cnt:0277d1216f80ae1adeed84a686ed34c9b2931fc2 kmod.c swh:1:cnt:57b939c81bce5d06fa587df8915f05affbe22b82 sched/deadline.c $ swh-identify --no-filename /usr/src/linux/kernel/ swh:1:dir:f9f858a48d663b3809c9e2f336412717496202ab ``` diff --git a/debian/changelog b/debian/changelog index d32f8ec..66bdb3c 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,1361 +1,1363 @@ -swh-model (6.5.0-1~swh1~bpo10+1) buster-swh; urgency=medium +swh-model (6.5.1-1~swh1) unstable-swh; urgency=medium - * Rebuild for buster-swh + * New upstream release 6.5.1 - (tagged by Antoine Lambert + on 2022-09-30 15:23:14 +0200) + * Upstream changes: - version 6.5.1 - -- Software Heritage autobuilder (on jenkins-debian1) Mon, 26 Sep 2022 16:17:36 +0000 + -- Software Heritage autobuilder (on jenkins-debian1) Fri, 30 Sep 2022 13:27:54 +0000 swh-model (6.5.0-1~swh1) unstable-swh; urgency=medium * New upstream release 6.5.0 - (tagged by Nicolas Dandrimont on 2022-09-26 18:11:53 +0200) * Upstream changes: - Release swh.model v6.5.0 - Bundle a lot of performance improvements in swh.model.model. -- Software Heritage autobuilder (on jenkins-debian1) Mon, 26 Sep 2022 16:16:05 +0000 swh-model (6.4.1-1~swh1) unstable-swh; urgency=medium * New upstream release 6.4.1 - (tagged by Valentin Lorentz on 2022-08-31 10:27:27 +0200) * Upstream changes: - v6.4.1 - * Add support for old dicts in RawExtrinsicMetadata.from_dict -- Software Heritage autobuilder (on jenkins-debian1) Wed, 31 Aug 2022 08:31:28 +0000 swh-model (6.4.0-1~swh1) unstable-swh; urgency=medium * New upstream release 6.4.0 - (tagged by Valentin Lorentz on 2022-08-12 10:58:05 +0200) * Upstream changes: - v6.4.0 - * hypothesis_strategies: Avoid generating timestamps which overflow datetime.datetime.fromtimestamp - * model: Allow unresolved branches when computing Snapshot hash -- Software Heritage autobuilder (on jenkins-debian1) Fri, 12 Aug 2022 09:01:41 +0000 swh-model (6.3.1-1~swh1) unstable-swh; urgency=medium * New upstream release 6.3.1 - (tagged by Valentin Lorentz on 2022-07-20 11:35:17 +0200) * Upstream changes: - v6.3.1 - * hypothesis_strategies: Prevent generation of invalid 'microseconds' values -- Software Heritage autobuilder (on jenkins-debian1) Wed, 20 Jul 2022 09:39:38 +0000 swh-model (6.3.0-1~swh1) unstable-swh; urgency=medium * New upstream release 6.3.0 - (tagged by Valentin Lorentz on 2022-07-12 10:46:01 +0200) * Upstream changes: - v6.3.0 - * Dedup `format_git_object_from_headers` - * hashutil: Add type annotations to conversion functions - * model: Add Directory.from_possibly_duplicated_entries factory - * hypothesis_strategies: Add kwargs to composite strategies, to replace default sub-strategies -- Software Heritage autobuilder (on jenkins-debian1) Tue, 12 Jul 2022 08:50:50 +0000 swh-model (6.2.0-1~swh1) unstable-swh; urgency=medium * New upstream release 6.2.0 - (tagged by Valentin Lorentz on 2022-04-27 18:36:27 +0200) * Upstream changes: - v6.2.0 - * Add missing `content_git_object` - * test/pre-commit maintenance -- Software Heritage autobuilder (on jenkins-debian1) Wed, 27 Apr 2022 16:40:00 +0000 swh-model (6.1.0-1~swh1) unstable-swh; urgency=medium * New upstream release 6.1.0 - (tagged by David Douard on 2022-04-11 12:43:26 +0200) * Upstream changes: - v6.1.0 -- Software Heritage autobuilder (on jenkins-debian1) Mon, 11 Apr 2022 10:47:57 +0000 swh-model (6.0.1-1~swh1) unstable-swh; urgency=medium * New upstream release 6.0.1 - (tagged by Antoine R. Dumont (@ardumont) on 2022-04-01 10:21:34 +0200) * Upstream changes: - v6.0.1 - Fix documentation papercuts -- Software Heritage autobuilder (on jenkins-debian1) Fri, 01 Apr 2022 08:25:24 +0000 swh-model (6.0.0-1~swh1) unstable-swh; urgency=medium * New upstream release 6.0.0 - (tagged by Valentin Lorentz on 2022-03-23 11:02:28 +0100) * Upstream changes: - v6.0.0 - * Add objects with non-None raw_manifest to TEST_OBJECTS - * Exclude name and email attributes from People comparison - * Add support for None as author or committer of a Revision -- Software Heritage autobuilder (on jenkins-debian1) Wed, 23 Mar 2022 10:06:05 +0000 swh-model (5.0.0-1~swh1) unstable-swh; urgency=medium * New upstream release 5.0.0 - (tagged by Valentin Lorentz on 2022-03-16 10:33:49 +0100) * Upstream changes: - v5.0.0 - * Fix f-string - * Fix crash in check_entries. - * Add missing __slots__ to HashableObjectWithManifest - * docs: Explain we prefer dir SWHIDs over rev/rel. - * Remove 'offset' and 'negative_utc' arguments and make them optional - * Remove deprecated property 'TimestampWithTimezone.offset' -- Software Heritage autobuilder (on jenkins-debian1) Wed, 16 Mar 2022 09:38:26 +0000 swh-model (4.4.0-1~swh1) unstable-swh; urgency=medium * New upstream release 4.4.0 - (tagged by Valentin Lorentz on 2022-01-21 14:08:57 +0100) * Upstream changes: - v4.4.0 - * model: Add support for more edge cases in _parse_offset_bytes - * model: Add method 'TimestampWithTimezone.offset_minutes' -- Software Heritage autobuilder (on jenkins-debian1) Fri, 21 Jan 2022 13:12:25 +0000 swh-model (4.3.0-1~swh1) unstable-swh; urgency=medium * New upstream release 4.3.0 - (tagged by Valentin Lorentz on 2022-01-14 15:10:34 +0100) * Upstream changes: - v4.3.0 - * docs: Add anchors to important sections of persistent-identifiers.rst - * Fix TimestampWithTimezone.from_dict() on datetimes before 1970 with non- integer seconds - * TimestampWithTimezone: Make 'offset' and 'negative_utc' optional -- Software Heritage autobuilder (on jenkins-debian1) Fri, 14 Jan 2022 14:13:48 +0000 swh-model (4.2.0-1~swh1) unstable-swh; urgency=medium * New upstream release 4.2.0 - (tagged by Valentin Lorentz on 2022-01-10 15:56:36 +0100) * Upstream changes: - v4.2.0 - * git_objects: Use raw offset_bytes to format dates, and remove format_offset() -- Software Heritage autobuilder (on jenkins-debian1) Mon, 10 Jan 2022 14:59:18 +0000 swh-model (4.1.0-1~swh1) unstable-swh; urgency=medium * New upstream release 4.1.0 - (tagged by Nicolas Dandrimont on 2021-12-22 15:58:36 +0100) * Upstream changes: - Release swh.model v4.1.0 - Drop pre-3.6 blake2 compatibility, which hasn't been in use since - we've mandated python3.7 anyway. -- Software Heritage autobuilder (on jenkins-debian1) Wed, 22 Dec 2021 15:01:40 +0000 swh-model (4.0.0-1~swh1) unstable-swh; urgency=medium * New upstream release 4.0.0 - (tagged by Valentin Lorentz on 2021-12-22 13:24:58 +0100) * Upstream changes: - v4.0.0 - * Add attribute TimestampWithTimezone.offset_bytes, to store raw Git offsets - * model: Add a check() method to model objects - * test_model: Fix compatibility with pytest-xdist - * docs: Update the data model description - * hypothesis_strategies: Generate only consistent directory entry permissions. - * model: Add a raw_manifest attribute -- Software Heritage autobuilder (on jenkins-debian1) Wed, 22 Dec 2021 12:28:54 +0000 swh-model (3.2.0-1~swh1) unstable-swh; urgency=medium * New upstream release 3.2.0 - (tagged by Valentin Lorentz on 2021-12-15 13:36:48 +0100) * Upstream changes: - v3.2.0 - * hypothesis_strategies: Ensure to generate valid directory entry name - * from_disk: Implement Directory.__contains__ -- Software Heritage autobuilder (on jenkins-debian1) Wed, 15 Dec 2021 12:39:37 +0000 swh-model (3.1.0-1~swh1) unstable-swh; urgency=medium * New upstream release 3.1.0 - (tagged by Antoine Lambert on 2021-12-06 19:35:40 +0100) * Upstream changes: - version 0.3.1 -- Software Heritage autobuilder (on jenkins-debian1) Mon, 06 Dec 2021 18:51:48 +0000 swh-model (3.0.0-1~swh1) unstable-swh; urgency=medium * New upstream release 3.0.0 - (tagged by Valentin Lorentz on 2021-09-28 15:59:18 +0200) * Upstream changes: - v3.0.0 - * Add bazaar as supported revision type - * Move SWHID classes and functions from identifiers.py to swhids.py - * Refactor identifiers & model to make *_git_object() functions work on model classes instead of dicts - * Move manifest computation functions from identifiers.py to git_objects.py - * Remove identifier_to_bytes and identifier_to_hex - * Deprecate identifiers.py -- Software Heritage autobuilder (on jenkins-debian1) Tue, 28 Sep 2021 14:05:19 +0000 swh-model (2.9.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.9.0 - (tagged by Valentin Lorentz on 2021-09-16 14:21:44 +0200) * Upstream changes: - v2.9.0 - * HashableObject: Add type annotation for 'id' attribute -- Software Heritage autobuilder (on jenkins-debian1) Thu, 16 Sep 2021 12:24:48 +0000 swh-model (2.8.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.8.0 - (tagged by Antoine R. Dumont (@ardumont) on 2021-07-27 16:20:14 +0200) * Upstream changes: - v2.8.0 - Add a CVS revision type for use with the CVS loader -- Software Heritage autobuilder (on jenkins-debian1) Tue, 27 Jul 2021 14:26:10 +0000 swh-model (2.7.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.7.0 - (tagged by Nicolas Dandrimont on 2021-07-23 16:50:59 +0200) * Upstream changes: - Release swh.model 2.7.0 - Add versioning of ExtID objects -- Software Heritage autobuilder (on jenkins-debian1) Fri, 23 Jul 2021 14:53:44 +0000 swh-model (2.6.4-1~swh1) unstable-swh; urgency=medium * New upstream release 2.6.4 - (tagged by Daniele Serafini on 2021-06-29 13:42:54 +0100) * Upstream changes: - make deduplication optional when iterating over the merkle tree -- Software Heritage autobuilder (on jenkins-debian1) Fri, 02 Jul 2021 16:11:31 +0000 swh-model (2.6.3-1~swh1) unstable-swh; urgency=medium * New upstream release 2.6.3 - (tagged by Valentin Lorentz on 2021-06-25 16:13:53 +0200) * Upstream changes: - v2.6.3 - * hypothesis_strategies: Generate None metadata instead of {} -- Software Heritage autobuilder (on jenkins-debian1) Fri, 25 Jun 2021 14:17:34 +0000 swh-model (2.6.2-1~swh1) unstable-swh; urgency=medium * New upstream release 2.6.2 - (tagged by Valentin Lorentz on 2021-06-25 12:40:45 +0200) * Upstream changes: - v2.6.2 - * from_disk: get swhid from Content/Directory objects - * hypothesis_strategies: Add raw_extrinsic_metadata() strategy -- Software Heritage autobuilder (on jenkins-debian1) Fri, 25 Jun 2021 10:44:34 +0000 swh-model (2.6.1-1~swh1) unstable-swh; urgency=medium * New upstream release 2.6.1 - (tagged by Antoine Lambert on 2021-06-16 11:58:53 +0200) * Upstream changes: - version 2.6.1 -- Software Heritage autobuilder (on jenkins-debian1) Wed, 16 Jun 2021 10:03:28 +0000 swh-model (2.6.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.6.0 - (tagged by David Douard on 2021-06-15 16:51:49 +0200) * Upstream changes: - v2.6.0 -- Software Heritage autobuilder (on jenkins-debian1) Tue, 15 Jun 2021 14:56:10 +0000 swh-model (2.5.1-1~swh1) unstable-swh; urgency=medium * New upstream release 2.5.1 - (tagged by David Douard on 2021-05-20 15:22:50 +0200) * Upstream changes: - v2.5.1 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 20 May 2021 13:40:27 +0000 swh-model (2.5.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.5.0 - (tagged by Valentin Lorentz on 2021-05-11 12:02:49 +0200) * Upstream changes: - v2.5.0 - * identifiers: Expose manifest/git_object computation -- Software Heritage autobuilder (on jenkins-debian1) Tue, 11 May 2021 10:07:47 +0000 swh-model (2.4.2-1~swh1) unstable-swh; urgency=medium * New upstream release 2.4.2 - (tagged by Valentin Lorentz on 2021-05-06 14:31:04 +0200) * Upstream changes: - v2.4.2 - * docs/persistent-identifiers: Add guidelines for fixing invalid SWHIDs. - * Blacklist attr 21.1.0 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 06 May 2021 12:35:43 +0000 swh-model (2.4.1-1~swh1) unstable-swh; urgency=medium * New upstream release 2.4.1 - (tagged by Antoine Lambert on 2021-04-29 14:19:28 +0200) * Upstream changes: - version 2.4.1 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 29 Apr 2021 12:23:21 +0000 swh-model (2.4.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.4.0 - (tagged by Antoine Lambert on 2021-04-13 15:26:51 +0200) * Upstream changes: - version 2.4.0 -- Software Heritage autobuilder (on jenkins-debian1) Tue, 13 Apr 2021 13:31:21 +0000 swh-model (2.3.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.3.0 - (tagged by Nicolas Dandrimont on 2021-03-19 17:15:00 +0100) * Upstream changes: - Release swh.model 2.3.0 - Properly truncate RawExtrinsicMetadata objects to a precision of one - second, as does their unique id. -- Software Heritage autobuilder (on jenkins-debian1) Fri, 19 Mar 2021 16:17:48 +0000 swh-model (2.2.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.2.0 - (tagged by Valentin Lorentz on 2021-03-15 10:32:36 +0100) * Upstream changes: - v2.2.0 - * Add a swhid() method to RawExtrinsicMetadata. -- Software Heritage autobuilder (on jenkins-debian1) Mon, 15 Mar 2021 09:35:25 +0000 swh-model (2.1.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.1.0 - (tagged by David Douard on 2021-03-11 14:19:00 +0100) * Upstream changes: - v2.1.0 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 11 Mar 2021 13:21:40 +0000 swh-model (2.0.0-1~swh1) unstable-swh; urgency=medium * New upstream release 2.0.0 - (tagged by Valentin Lorentz on 2021-03-05 10:11:47 +0100) * Upstream changes: - v2.0.0 - Breaking change: - * model: Remove override of RawExtrinsicMetadata.unique_key(), so it now returns the hash. - Other changes: - * identifiers: Add raw_extrinsic_metadata_identifier - * model: Add 'id' field to RawExtrinsicMetadata -- Software Heritage autobuilder (on jenkins-debian1) Fri, 05 Mar 2021 09:14:35 +0000 swh-model (1.0.1-1~swh1) unstable-swh; urgency=medium * New upstream release 1.0.1 - (tagged by Valentin Lorentz on 2021-03-04 15:08:55 +0100) * Upstream changes: - v1.0.1 - * cli: stop using the deprecated SWHID class - * identifiers: Remove the deprecated SWHID class -- Software Heritage autobuilder (on jenkins-debian1) Thu, 04 Mar 2021 14:11:09 +0000 swh-model (1.0.0-1~swh1) unstable-swh; urgency=medium * New upstream release 1.0.0 - (tagged by Valentin Lorentz on 2021-03-01 18:01:29 +0100) * Upstream changes: - v1.0.0 - Two breaking changes: - * RawExtrinsicMetadata: Use ExtendedSWHID as target and remove type - * RawExtrinsicMetadata: Use CoreSWHID instead of SWHID for contexts - And two minor changes: - * Add CoreSWHID.to_extended() - * Add a swhid() method to all hashable objects. -- Software Heritage autobuilder (on jenkins-debian1) Tue, 02 Mar 2021 08:18:42 +0000 swh-model (0.13.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.13.0 - (tagged by Valentin Lorentz on 2021-02-25 17:56:31 +0100) * Upstream changes: - v0.13.0 - * Update persistent identifiers doc with pip install info - * Make explicit Python 3 dependency - * tests: Clean hashutil._blake2_hash_cache after mocking blake2 functions. - * Introduce new classes CoreSWHID/QualifiedSWHID/ExtendedSWHID - * Deprecate SWHID class - * Disallow 'ori' type in SWHID class -- Software Heritage autobuilder (on jenkins-debian1) Thu, 25 Feb 2021 16:59:26 +0000 swh-model (0.12.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.12.0 - (tagged by David Douard on 2021-01-26 17:22:28 +0100) * Upstream changes: - v0.12.0 -- Software Heritage autobuilder (on jenkins-debian1) Tue, 26 Jan 2021 16:27:16 +0000 swh-model (0.11.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.11.0 - (tagged by Antoine R. Dumont (@ardumont) on 2021-01-20 15:31:54 +0100) * Upstream changes: - v0.11.0 - model: Allow new status values not_found and failed to OriginVisitStatus -- Software Heritage autobuilder (on jenkins-debian1) Wed, 20 Jan 2021 14:34:53 +0000 swh-model (0.10.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.10.0 - (tagged by Vincent SELLIER on 2021-01-14 14:13:22 +0100) * Upstream changes: - v0.10.0 - * 2021-01-12 Add an optional type field on OriginVisitStatus object - * 2021-01-12 test_identifiers: Reorder SWHID tests. - * 2021-01-12 test_identifiers: Make sure that {directory,revision,release,snapshot}_identifier() doesn't just return a value from the dict. - * 2021-01-04 Add missing slots=True for Directory. - * 2020-12-19 SWHID parsing: simplify and deduplicate validation logic - * 2020-12-14 model: Make all classes slotted. -- Software Heritage autobuilder (on jenkins-debian1) Thu, 14 Jan 2021 13:16:10 +0000 swh-model (0.9.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.9.0 - (tagged by Nicolas Dandrimont on 2020-11-17 10:12:47 +0100) * Upstream changes: - Release swh.model v0.9.0 - Drop backwards compatibility for RawExtrinsicMetadata.id -- Software Heritage autobuilder (on jenkins-debian1) Tue, 17 Nov 2020 09:15:43 +0000 swh-model (0.8.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.8.0 - (tagged by Antoine R. Dumont (@ardumont) on 2020-11-12 13:09:48 +0100) * Upstream changes: - v0.8.0 - identifiers.parse_swhid: Make SWHIDs with whitespaces invalid - identifiers.parse_swhid: Check the swhid qualifiers and fail if invalid - model.identifiers: Improve error messages in case of invalid SWHIDs -- Software Heritage autobuilder (on jenkins-debian1) Thu, 12 Nov 2020 12:10:46 +0000 swh-model (0.7.3-1~swh1) unstable-swh; urgency=medium * New upstream release 0.7.3 - (tagged by Nicolas Dandrimont on 2020-10-27 16:03:58 +0100) * Upstream changes: - Release swh.model v0.7.3 - Reduce the amount of DeprecationWarnings for RawExtrinsicMetadata -- Software Heritage autobuilder (on jenkins-debian1) Tue, 27 Oct 2020 15:06:50 +0000 swh-model (0.7.2-1~swh1) unstable-swh; urgency=medium * New upstream release 0.7.2 - (tagged by Nicolas Dandrimont on 2020-10-27 10:37:19 +0100) * Upstream changes: - Release swh.model v0.7.2 - Add a new -- exclude flag to swh identify - Migrate RawExtrinsicMetadata `id` attribute to `target` - Future-proof the swh.model.model.HashableObject interface -- Software Heritage autobuilder (on jenkins-debian1) Tue, 27 Oct 2020 09:41:19 +0000 swh-model (0.7.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.7.1 - (tagged by Valentin Lorentz on 2020-10-12 12:16:47 +0200) * Upstream changes: - v0.7.1 - Add a 'unique_key' method on model objects -- Software Heritage autobuilder (on jenkins-debian1) Mon, 12 Oct 2020 10:19:10 +0000 swh-model (0.7.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.7.0 - (tagged by Antoine R. Dumont (@ardumont) on 2020-10-07 12:47:02 +0200) * Upstream changes: - v0.7.0 - cli: make SWHIDParamType return SWHID type instead of string - tox.ini: pin black to the pre- commit version (19.10b0) to avoid flip-flops - Merge the two test_identifiers.py files. -- Software Heritage autobuilder (on jenkins-debian1) Wed, 07 Oct 2020 10:47:55 +0000 swh-model (0.6.7-3~swh1) unstable-swh; urgency=medium * Fix a typo in d/control. -- David Douard Fri, 25 Sep 2020 17:36:14 +0200 swh-model (0.6.7-2~swh1) unstable-swh; urgency=medium * Fix dependencies on d/control -- David Douard Fri, 25 Sep 2020 17:03:31 +0200 swh-model (0.6.7-1~swh1) unstable-swh; urgency=medium * New upstream release 0.6.7 - (tagged by David Douard on 2020-09-25 15:28:58 +0200) * Upstream changes: - v0.6.7 -- Software Heritage autobuilder (on jenkins-debian1) Fri, 25 Sep 2020 13:32:18 +0000 swh-model (0.6.6-1~swh1) unstable-swh; urgency=medium * New upstream release 0.6.6 - (tagged by Antoine R. Dumont (@ardumont) on 2020-08-07 10:18:50 +0200) * Upstream changes: - v0.6.6 - model.Content.to_dict: Remove ctime entry when it's None - model: Add Sha1 alias -- Software Heritage autobuilder (on jenkins-debian1) Fri, 07 Aug 2020 08:22:35 +0000 swh-model (0.6.5-1~swh1) unstable-swh; urgency=medium * New upstream release 0.6.5 - (tagged by Antoine R. Dumont (@ardumont) on 2020-08-06 19:59:26 +0200) * Upstream changes: - v0.6.5 - model: Add final object_type field on metadata related model objects -- Software Heritage autobuilder (on jenkins-debian1) Thu, 06 Aug 2020 18:01:05 +0000 swh-model (0.6.4-1~swh1) unstable-swh; urgency=medium * New upstream release 0.6.4 - (tagged by Antoine R. Dumont (@ardumont) on 2020-08-06 18:44:48 +0200) * Upstream changes: - v0.6.4 - Use correct setuptools-scm keyword this time -- Software Heritage autobuilder (on jenkins-debian1) Thu, 06 Aug 2020 16:47:14 +0000 swh-model (0.6.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.6.1 - (tagged by Valentin Lorentz on 2020-07-31 10:47:56 +0200) * Upstream changes: - v0.6.1 - * Declare pytest markers - * Import Mapping from collections.abc instead of collections - * Fix incorrectly typed null constants in extra_headers byte strings - * add ImmutableDict.__repr__ - * Add missing object_type class attributes on MetadataAuthority, MetadataFetcher, and RawExtrinsicMetadata. -- Software Heritage autobuilder (on jenkins-debian1) Fri, 31 Jul 2020 08:51:42 +0000 swh-model (0.6.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.6.0 - (tagged by Valentin Lorentz on 2020-07-20 12:45:59 +0200) * Upstream changes: - v0.6.0 - * Rework dia -> pdf pipeline for inkscape 1.0 - * Rename MetadataAuthorityType.DEPOSIT to MetadataAuthorityType.DEPOSIT_CLIENT. -- Software Heritage autobuilder (on jenkins-debian1) Mon, 20 Jul 2020 10:49:27 +0000 swh-model (0.5.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.5.0 - (tagged by Antoine Lambert on 2020-07-08 17:12:44 +0200) * Upstream changes: - version 0.5.0 -- Software Heritage autobuilder (on jenkins-debian1) Wed, 08 Jul 2020 15:23:51 +0000 swh-model (0.4.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.4.0 - (tagged by David Douard on 2020-07-06 14:13:31 +0200) * Upstream changes: - v0.4.0 -- Software Heritage autobuilder (on jenkins-debian1) Mon, 06 Jul 2020 12:16:51 +0000 swh-model (0.3.8-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.8 - (tagged by Antoine Lambert on 2020-07-03 16:06:44 +0200) * Upstream changes: - version 0.3.8 -- Software Heritage autobuilder (on jenkins-debian1) Fri, 03 Jul 2020 14:10:51 +0000 swh-model (0.3.7-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.7 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-02 15:15:46 +0200) * Upstream changes: - v0.3.7 - Refactor common loader behavior within from_disk.iter_directory - Unify object_type some more within the merkle and from_disk modules -- Software Heritage autobuilder (on jenkins-debian1) Thu, 02 Jul 2020 13:17:32 +0000 swh-model (0.3.6-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.6 - (tagged by Antoine R. Dumont (@ardumont) on 2020-07-01 15:46:23 +0200) * Upstream changes: - v0.3.6 - model.OriginVisit: Drop obsolete fields -- Software Heritage autobuilder (on jenkins-debian1) Wed, 01 Jul 2020 13:48:43 +0000 swh-model (0.3.5-2~swh1) unstable-swh; urgency=medium * Update dependency + Bump -- Antoine R. Dumont (@ardumont) Tue, 30 Jun 2020 12:40:52 +0200 swh-model (0.3.5-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.5 - (tagged by Antoine R. Dumont (@ardumont) on 2020-06-30 11:21:07 +0200) * Upstream changes: - v0.3.5 - Tag model entities with their "object_type" -- Software Heritage autobuilder (on jenkins-debian1) Tue, 30 Jun 2020 09:31:43 +0000 swh-model (0.3.4-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.4 - (tagged by Antoine R. Dumont (@ardumont) on 2020-06-24 10:43:48 +0200) * Upstream changes: - v0.3.4 - OriginVisitStatus: Allow "created" status - model.OriginVisit: Make obsolete fields optional - swh.model.model.OriginVisit: Drop the dateutil.parser.parse use -- Software Heritage autobuilder (on jenkins-debian1) Wed, 24 Jun 2020 08:47:12 +0000 swh-model (0.3.3-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.3 - (tagged by Antoine R. Dumont (@ardumont) on 2020-06-17 09:38:34 +0200) * Upstream changes: - v0.3.3 - model.hypothesis_strategies: Make metadata always none on origin_visit -- Software Heritage autobuilder (on jenkins-debian1) Wed, 17 Jun 2020 07:40:50 +0000 swh-model (0.3.2-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.2 - (tagged by David Douard on 2020-06-16 10:41:05 +0200) * Upstream changes: - v0.3.2 -- Software Heritage autobuilder (on jenkins-debian1) Tue, 16 Jun 2020 08:45:55 +0000 swh-model (0.3.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.1 - (tagged by David Douard on 2020-06-15 09:43:30 +0200) * Upstream changes: - v0.3.1 -- Software Heritage autobuilder (on jenkins-debian1) Mon, 15 Jun 2020 07:52:09 +0000 swh-model (0.3.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.3.0 - (tagged by David Douard on 2020-06-03 11:59:02 +0200) * Upstream changes: - v0.3.0 -- Software Heritage autobuilder (on jenkins-debian1) Wed, 03 Jun 2020 10:04:35 +0000 swh-model (0.2.2-1~swh1) unstable-swh; urgency=medium * New upstream release 0.2.2 - (tagged by David Douard on 2020-06-03 11:28:38 +0200) * Upstream changes: - v0.2.2 -- Software Heritage autobuilder (on jenkins-debian1) Wed, 03 Jun 2020 09:33:46 +0000 swh-model (0.2.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.2.1 - (tagged by David Douard on 2020-05-29 17:39:37 +0200) * Upstream changes: - v0.2.1 -- Software Heritage autobuilder (on jenkins-debian1) Fri, 29 May 2020 15:43:44 +0000 swh-model (0.2.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.2.0 - (tagged by David Douard on 2020-05-25 10:06:12 +0200) * Upstream changes: - v0.2.0 -- Software Heritage autobuilder (on jenkins-debian1) Mon, 25 May 2020 08:11:07 +0000 swh-model (0.1.1-1~swh1) unstable-swh; urgency=medium * New upstream release 0.1.1 - (tagged by Antoine R. Dumont (@ardumont) on 2020-05-05 14:43:40 +0200) * Upstream changes: - v0.1.1 - Make aware_datetimes() generate only ISO8601-encodable datetimes -- Software Heritage autobuilder (on jenkins-debian1) Tue, 05 May 2020 12:45:37 +0000 swh-model (0.1.0-1~swh1) unstable-swh; urgency=medium * New upstream release 0.1.0 - (tagged by Stefano Zacchiroli on 2020-04-30 19:23:13 +0200) * Upstream changes: - v0.1.0 / 2020-04-30 - * SWHID spec: full reread - * setup.py: add documentation link - * hypothesis_strategies: Generate aware datetimes instead of naive ones. - * doc: check-in IANA registration template for the "swh" URI scheme - * Restructure SWHID documentation in preparation for T2385 - merge grammars into a single one - explain better that SWHIDs are made up of core identifier + qualifiers - separate qualifier into context and fragment onex - add reference to swh-identify -- Software Heritage autobuilder (on jenkins-debian1) Thu, 30 Apr 2020 20:31:00 +0000 swh-model (0.0.69-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.69 - (tagged by Stefano Zacchiroli on 2020-04-28 16:05:58 +0200) * Upstream changes: - v0.0.69 / 2020-04-28 - * SWHID spec: bump version to 1.3 and add last modified date - * SWHID spec: make SWHIDs plural where needed - * SWHID spec: simplify and generalize escaping requirements - * SWHID spec: add support for IRI - * SWHID: deal with escaping in origin qualifiers - * SWHID doc: improve wording of intrinsic parts v. the rest -- Software Heritage autobuilder (on jenkins-debian1) Tue, 28 Apr 2020 14:10:35 +0000 swh-model (0.0.68-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.68 - (tagged by David Douard on 2020-04-21 16:20:58 +0200) * Upstream changes: - v0.0.68 -- Software Heritage autobuilder (on jenkins-debian1) Tue, 21 Apr 2020 14:28:38 +0000 swh-model (0.0.67-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.67 - (tagged by Stefano Zacchiroli on 2020-04-17 17:49:42 +0200) * Upstream changes: - v0.0.67 / 2020-04-17 - * CLI: add test for swh identify w/o args - * CLI: require explicit "-" to identify via stdin - * SWHID doc: fix minor grammar issue - * SWHID doc: fix link in CISE paper reference - * identifiers.py: reference to SWHIDs using explicit anchors - * swh identify: embrace SWHID naming in user-facing doc/messages - * PID doc: embrace the SWHID naming - * PID doc: add reference to CISE paper - * doc: document identify CLI -- Software Heritage autobuilder (on jenkins-debian1) Fri, 17 Apr 2020 15:54:03 +0000 swh-model (0.0.66-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.66 - (tagged by Antoine R. Dumont (@ardumont) on 2020-04-10 16:46:31 +0200) * Upstream changes: - v0.0.66 - rename-visit-status model: Rename OriginVisitUpdate to OriginVisitStatus -- Software Heritage autobuilder (on jenkins-debian1) Fri, 10 Apr 2020 14:48:17 +0000 swh-model (0.0.65-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.65 - (tagged by Antoine R. Dumont (@ardumont) on 2020-04-09 16:25:24 +0200) * Upstream changes: - v0.0.65 - from_disk: path parameter to dir_filter functions - Enable black -- Software Heritage autobuilder (on jenkins-debian1) Thu, 09 Apr 2020 14:27:21 +0000 swh-model (0.0.64-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.64 - (tagged by Antoine Lambert on 2020-04-03 15:00:36 +0200) * Upstream changes: - version 0.0.64 -- Software Heritage autobuilder (on jenkins-debian1) Fri, 03 Apr 2020 13:03:34 +0000 swh-model (0.0.63-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.63 - (tagged by Antoine R. Dumont (@ardumont) on 2020-04-01 10:07:07 +0200) * Upstream changes: - v0.0.63 - origin/master model: Add new OriginVisitUpdate model object + test strategy - docs: Extend SWH PID definition with additional context qualifiers. -- Software Heritage autobuilder (on jenkins-debian1) Wed, 01 Apr 2020 08:08:58 +0000 swh-model (0.0.62-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.62 - (tagged by Valentin Lorentz on 2020-03-26 14:19:40 +0100) * Upstream changes: - v0.0.62 - * identifiers: encode origin URLs in utf-8 -- Software Heritage autobuilder (on jenkins-debian1) Thu, 26 Mar 2020 13:22:20 +0000 swh-model (0.0.60-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.60 - (tagged by Valentin Lorentz on 2020-03-05 12:05:18 +0100) * Upstream changes: - v0.0.60 - * Add a method to generate Content/SkippedContent from binary data - * Draw contents from a byte string instead of generating arbitrary hashes - * Add classmethod Person.from_address, to parse from 'name ' strings. -- Software Heritage autobuilder (on jenkins-debian1) Thu, 05 Mar 2020 11:07:50 +0000 swh-model (0.0.59-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.59 - (tagged by Nicolas Dandrimont on 2020-02-27 18:03:53 +0100) * Upstream changes: - Release swh.model v0.0.59 - Use proper hypothesis strategy to generate Person objects -- Software Heritage autobuilder (on jenkins-debian1) Thu, 27 Feb 2020 17:07:16 +0000 swh-model (0.0.57-1~swh2) unstable-swh; urgency=medium * Bump dependency release -- Antoine R. Dumont (@ardumont) Thu, 27 Feb 2020 16:24:21 +0200 swh-model (0.0.57-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.57 - (tagged by Valentin Lorentz on 2020-02-27 15:17:04 +0100) * Upstream changes: - v0.0.57 - * Add method BaseModel.hashes(). - * Re-introduce the swh.core dependency in swh.model[cli] - * Add support for skipping large contents in from_disk. - * Add to_model() method to from_disk.{Content,Directory}, to convert to canonical model objects. - * Take the value of MerkleNode.data into account to compute equality. - * Add method MerkleNode.iter_tree, to visit all nodes in the subtree of a node. - * Add from_datetime and from_iso8601 constructors for TimestampWithTimezone. - * Make attributes name and email of Person optional. -- Software Heritage autobuilder (on jenkins-debian1) Thu, 27 Feb 2020 14:20:21 +0000 swh-model (0.0.56-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.56 - (tagged by Valentin Lorentz on 2020-02-10 11:46:35 +0100) * Upstream changes: - v0.0.56 - Make OriginVisit.snapshot optional. -- Software Heritage autobuilder (on jenkins-debian1) Mon, 10 Feb 2020 10:48:55 +0000 swh-model (0.0.55-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.55 - (tagged by Valentin Lorentz on 2020-02-07 16:13:23 +0100) * Upstream changes: - v0.0.55 - * Make content length mandatory. - * Make 'visible' the default status for present Contents. -- Software Heritage autobuilder (on jenkins-debian1) Fri, 07 Feb 2020 15:16:58 +0000 swh-model (0.0.54-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.54 - (tagged by Valentin Lorentz on 2020-02-06 13:15:45 +0100) * Upstream changes: - v0.0.54 - * Split Content class into two classes, for missing and non-missing contents. -- Software Heritage autobuilder (on jenkins-debian1) Thu, 06 Feb 2020 12:18:04 +0000 swh-model (0.0.53-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.53 - (tagged by Valentin Lorentz on 2020-02-03 15:58:31 +0100) * Upstream changes: - v0.0.53 - * hypothesis_strategies/snapshots: Explain last post-processing step - * cli: add support for reading a file content from stdin in 'swh identify' command - * model: Update revision date types to be optional -- Software Heritage autobuilder (on jenkins-debian1) Mon, 03 Feb 2020 15:01:26 +0000 swh-model (0.0.52-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.52 - (tagged by Antoine Lambert on 2019-11-29 16:27:24 +0100) * Upstream changes: - version 0.0.52 -- Software Heritage autobuilder (on jenkins-debian1) Fri, 29 Nov 2019 15:30:57 +0000 swh-model (0.0.51-1~swh3) unstable-swh; urgency=medium * Add manual pytz dependency -- Nicolas Dandrimont Wed, 30 Oct 2019 17:52:33 +0100 swh-model (0.0.51-1~swh2) unstable-swh; urgency=medium * Add missing build-dependency on pytz -- Nicolas Dandrimont Wed, 30 Oct 2019 17:25:55 +0100 swh-model (0.0.51-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.51 - (tagged by Valentin Lorentz on 2019-10-30 15:03:19 +0100) * Upstream changes: - v0.0.51 - Make OriginVisit.origin a string instead of a dict. -- Software Heritage autobuilder (on jenkins-debian1) Wed, 30 Oct 2019 14:05:55 +0000 swh-model (0.0.50-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.50 - (tagged by David Douard on 2019-10-30 09:30:17 +0100) * Upstream changes: - v0.0.50 -- Software Heritage autobuilder (on jenkins-debian1) Wed, 30 Oct 2019 08:32:50 +0000 swh-model (0.0.49-1~swh2) unstable-swh; urgency=medium * Add missing dependency on dulwich for tests -- Nicolas Dandrimont Wed, 23 Oct 2019 14:37:45 +0200 swh-model (0.0.49-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.49 - (tagged by Nicolas Dandrimont on 2019-10-23 14:28:01 +0200) * Upstream changes: - Release swh.model v0.0.49 - Add symbolic refs to swh identify -t snapshot - Cleanup model.BaseModel.to_dict() recursion -- Software Heritage autobuilder (on jenkins-debian1) Wed, 23 Oct 2019 12:30:41 +0000 swh-model (0.0.48-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.48 - (tagged by Nicolas Dandrimont on 2019-10-18 17:06:59 +0200) * Upstream changes: - Release swh.model 0.0.48 - Split CLI dependencies to another subpackage - Stop exporting origin.type in models - Document origin PIDs -- Software Heritage autobuilder (on jenkins-debian1) Fri, 18 Oct 2019 15:11:01 +0000 swh-model (0.0.47-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.47 - (tagged by Stefano Zacchiroli on 2019-09-27 10:20:40 +0200) * Upstream changes: - v0.0.47 - init.py: switch to documented way of extending path -- Software Heritage autobuilder (on jenkins-debian1) Fri, 27 Sep 2019 08:22:54 +0000 swh-model (0.0.46-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.46 - (tagged by Stefano Zacchiroli on 2019-09-20 15:51:17 +0200) * Upstream changes: - v0.0.46 - MANIFEST.in: ship py.typed -- Software Heritage autobuilder (on jenkins-debian1) Fri, 20 Sep 2019 13:53:45 +0000 swh-model (0.0.45-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.45 - (tagged by Stefano Zacchiroli on 2019-09-20 15:09:47 +0200) * Upstream changes: - v0.0.45 - * identifiers.py: do not inherit from on-the-fly namedtuple - * mypy: ignore django- stubs, needed only by hypothesis - * mypy.ini: remove left-over sample section - * typing: minimal changes to make a no-op mypy run pass - * fix indentation and spelling: make "make check" happy -- Software Heritage autobuilder (on jenkins-debian1) Fri, 20 Sep 2019 13:12:10 +0000 swh-model (0.0.44-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.44 - (tagged by Valentin Lorentz on 2019-09-04 14:36:01 +0200) * Upstream changes: - Fix Revision.from_dict to allow optional fields. -- Software Heritage autobuilder (on jenkins-debian1) Wed, 04 Sep 2019 13:07:59 +0000 swh-model (0.0.43-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.43 - (tagged by Antoine R. Dumont (@ardumont) on 2019-09-03 14:04:44 +0200) * Upstream changes: - v0.0.43 - swh identify: add support for origin PIDs - identifiers.py: add constants for 'swh:1' and sanitize namespace -- Software Heritage autobuilder (on jenkins-debian1) Tue, 03 Sep 2019 12:09:04 +0000 swh-model (0.0.42-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.42 - (tagged by Valentin Lorentz on 2019-08-22 14:04:03 +0200) * Upstream changes: - v0.0.42 - Tweak swh.model.model to be closer to what swh-storage - accepts for releases and origin visits. -- Software Heritage autobuilder (on jenkins-debian1) Thu, 22 Aug 2019 12:12:22 +0000 swh-model (0.0.41-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.41 - (tagged by Valentin Lorentz on 2019-08-20 11:46:13 +0200) * Upstream changes: - tweaks to swh.model.model to support more valid inputs - * Allow -1 as Content length. - * Add optional 'ctime' field to Content. - * Generated content with status=hidden should have a data field. - * Add a get_hash helper method to Content. -- Software Heritage autobuilder (on jenkins-debian1) Tue, 20 Aug 2019 09:50:09 +0000 swh-model (0.0.40-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.40 - (tagged by Valentin Lorentz on 2019-08-06 14:36:37 +0200) * Upstream changes: - Add SHA1_SIZE constant. -- Software Heritage autobuilder (on jenkins-debian1) Tue, 06 Aug 2019 12:38:36 +0000 swh-model (0.0.39-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.39 - (tagged by Valentin Lorentz on 2019-07-18 12:28:42 +0200) * Upstream changes: - * fix pyblake2 dependency * origin persistent identifiers * release metadata -- Software Heritage autobuilder (on jenkins-debian1) Thu, 18 Jul 2019 10:31:00 +0000 swh-model (0.0.38-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.38 - (tagged by Valentin Lorentz on 2019-06-18 13:40:20 +0200) * Upstream changes: - Remove dependency on swh-core. - This is a fix to workaround pip's inability to correctly solve - extra requirements (swh-model depends on swh-core[], but if other - packages depend on swh-model and swh-core[http], the 'http' extra - does not always get installed). -- Software Heritage autobuilder (on jenkins-debian1) Tue, 18 Jun 2019 11:50:14 +0000 swh-model (0.0.37-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.37 - (tagged by David Douard on 2019-05-15 15:44:21 +0200) * Upstream changes: - cli: add support for --help on the 'identify' cli tool -- Software Heritage autobuilder (on jenkins-debian1) Thu, 13 Jun 2019 14:40:16 +0000 swh-model (0.0.36-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.36 - (tagged by Valentin Lorentz on 2019-04-26 13:33:29 +0200) * Upstream changes: - Prevent from_dict() from changing its input dict. -- Software Heritage autobuilder (on jenkins-debian1) Fri, 26 Apr 2019 11:57:45 +0000 swh-model (0.0.35-1~swh2) unstable-swh; urgency=medium * Remove hypothesis directory -- Nicolas Dandrimont Thu, 18 Apr 2019 18:27:33 +0200 swh-model (0.0.35-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.35 - (tagged by Nicolas Dandrimont on 2019-04-11 12:05:11 +0200) * Upstream changes: - Release swh.model v0.0.35 - Fix hypothesis strategies to work in non-UTC timezones -- Software Heritage autobuilder (on jenkins-debian1) Thu, 11 Apr 2019 10:08:14 +0000 swh-model (0.0.34-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.34 - (tagged by Valentin Lorentz on 2019-04-09 18:30:50 +0200) * Upstream changes: - Limit Content.length to what the pgsql storage supports. -- Software Heritage autobuilder (on jenkins-debian1) Wed, 10 Apr 2019 07:45:31 +0000 swh-model (0.0.33-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.33 - (tagged by Valentin Lorentz on 2019-04-08 21:46:28 +0200) * Upstream changes: - Tune the model generation to work with the pgsql storage. -- Software Heritage autobuilder (on jenkins-debian1) Tue, 09 Apr 2019 15:11:51 +0000 swh-model (0.0.32-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.32 - (tagged by Valentin Lorentz on 2019-04-05 19:15:16 +0200) * Upstream changes: - Add a model based using 'attrs' and Hypothesis strategies to generate it. -- Software Heritage autobuilder (on jenkins-debian1) Mon, 08 Apr 2019 12:57:45 +0000 swh-model (0.0.31-1~swh2) unstable-swh; urgency=medium * Add new dependencies on python3-attr and python3-hypothesis -- Nicolas Dandrimont Mon, 08 Apr 2019 14:55:50 +0200 swh-model (0.0.31-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.31 - (tagged by Valentin Lorentz on 2019-04-04 20:46:15 +0200) * Upstream changes: - Make snapshot_identifier add the cycle to the exception's arguments when it detects one. -- Software Heritage autobuilder (on jenkins-debian1) Fri, 05 Apr 2019 09:07:35 +0000 swh-model (0.0.30-1~swh1) unstable-swh; urgency=medium * New upstream release 0.0.30 - (tagged by David Douard on 2019-01-08 12:28:35 +0100) * Upstream changes: - v0.0.30 -- Software Heritage autobuilder (on jenkins-debian1) Wed, 09 Jan 2019 17:31:53 +0000 swh-model (0.0.29-1~swh1) unstable-swh; urgency=medium * Release swh.model v0.0.29 * Reference iPRES paper in PID documentation * Remove deprecated swh.model.hashutil.hash_* functions * Split debian packaging to separate branch -- Nicolas Dandrimont Wed, 31 Oct 2018 18:26:32 +0100 swh-model (0.0.28-1~swh1) unstable-swh; urgency=medium * v0.0.28 * setup: prepare for pypi upload * tests: Initialize tox use * tests: Migrate to pytest * docs: Improve basic repository information * docs: document PID resolution possibilities other than Web UI / * hashutil: Migrate towards MultiHash api -- Antoine R. Dumont (@ardumont) Tue, 23 Oct 2018 16:24:21 +0200 swh-model (0.0.27-1~swh1) unstable-swh; urgency=medium * v0.0.27 * Refactor: Add MultiHash class to improve hash computations * swh.model.hashutil: Improve and clarify docstrings * swh.model.hashutil: Mark hash_* function as deprecated -- Antoine R. Dumont (@ardumont) Mon, 17 Sep 2018 12:07:59 +0200 swh-model (0.0.26-1~swh1) unstable-swh; urgency=medium * v0.0.26 * swh.model.identifiers: Open metadata in persistent_identifier method * refactor CLI tests to avoid duplicate assertion pairs * swh-identify: follow symlinks for CLI arguments (by default) * cli.py: prefer os.fsdecode() over manual fiddling with locale.getpref... * swh-identify: add support for passing multiple CLI arguments -- Antoine R. Dumont (@ardumont) Mon, 23 Jul 2018 14:29:54 +0200 swh-model (0.0.25-1~swh1) unstable-swh; urgency=medium * version 0.0.25 -- Antoine Lambert Fri, 29 Jun 2018 11:49:25 +0200 swh-model (0.0.24-1~swh1) unstable-swh; urgency=medium * v0.0.24 * swh.model.cli: Catch specific exception during identifiers check * identifiers: Validate input * identifiers: Raise when error during parsing persistent identifiers * Update blake2 support to be less Debian-specific * add swh-identify CLI tool to compute persistent identifiers * docs: Update high-level documentation (Merkle DAG description, * contextual information for persistent IDs, etc...) -- Antoine R. Dumont (@ardumont) Fri, 22 Jun 2018 15:38:32 +0200 swh-model (0.0.23-1~swh1) unstable-swh; urgency=medium * version 0.0.23 -- Antoine Lambert Tue, 29 May 2018 14:08:45 +0200 swh-model (0.0.22-1~swh1) unstable-swh; urgency=medium * version 0.0.22 -- Antoine Pietri Tue, 30 Jan 2018 18:22:42 +0100 swh-model (0.0.21-1~swh1) unstable-swh; urgency=medium * v0.0.21 * swh.model.identifiers: Add persistent identifier function * docs: document the naming scheme for persistent identifiers * bin/swh-hash-file: new binary to compute SWH-style content identifiers -- Antoine R. Dumont (@ardumont) Wed, 17 Jan 2018 11:06:33 +0100 swh-model (0.0.20-1~swh1) unstable-swh; urgency=medium * v0.0.20 * swh.model.hashutil.hash_data: Optionally integrate length in result * hashutil: add `snapshot` object type for git hashes * docs: add absolute anchor to documentation index -- Antoine R. Dumont (@ardumont) Wed, 20 Dec 2017 10:47:10 +0100 swh-model (0.0.19-1~swh1) unstable-swh; urgency=medium * Release swh.model version 0.0.19 * Update packaging runes -- Nicolas Dandrimont Thu, 12 Oct 2017 18:07:59 +0200 swh-model (0.0.18-1~swh1) unstable-swh; urgency=medium * Release swh.model v0.0.18 * Replace swh.model.git with swh.model.from_disk (T709). * Clean up documentation -- Nicolas Dandrimont Thu, 05 Oct 2017 20:48:29 +0200 swh-model (0.0.17-1~swh1) unstable-swh; urgency=medium * Release swh.model v0.0.17 * Clean up pyblake2 requirement for Python 3.5+ -- Nicolas Dandrimont Mon, 26 Jun 2017 14:41:49 +0200 swh-model (0.0.16-1~swh1) unstable-swh; urgency=medium * Release swh.model v0.0.16 * Make sure we generate proper permissions in directories -- Nicolas Dandrimont Fri, 07 Apr 2017 14:32:34 +0200 swh-model (0.0.15-1~swh1) unstable-swh; urgency=medium * v0.0.15 * Add possibility to compute new blake2 hashes * Add blake2s256 hash as default new hash computation algorithm -- Antoine R. Dumont (@ardumont) Fri, 24 Mar 2017 16:32:35 +0100 swh-model (0.0.14-1~swh1) unstable-swh; urgency=medium * v0.0.14 * Migrate functions from swh.core.hashutil to swh.model.hashutil -- Antoine R. Dumont (@ardumont) Wed, 15 Mar 2017 16:00:56 +0100 swh-model (0.0.13-1~swh1) unstable-swh; urgency=medium * Release swh.model v0.0.13 * Timestamps are now fully integer values -- Nicolas Dandrimont Tue, 14 Feb 2017 19:32:24 +0100 swh-model (0.0.12-1~swh1) unstable-swh; urgency=medium * Release swh.model v0.0.12 * Add more tests to git tree hash computations -- Nicolas Dandrimont Tue, 14 Jun 2016 17:08:20 +0200 swh-model (0.0.11-1~swh1) unstable-swh; urgency=medium * v0.0.11 * Open git.children_hashes api * Rename git.walk_and_compute_sha1_from_directory_2 to git.compute_hashes_from_directory * Remove dead code -- Antoine R. Dumont (@ardumont) Sat, 11 Jun 2016 02:23:19 +0200 swh-model (0.0.10-1~swh1) unstable-swh; urgency=medium * v0.0.10 * Add objects_per_type api * Open a new walk_and_compute_sha1_from_directory_2 api * Improve internal api regarding directory and tree hash computations -- Antoine R. Dumont (@ardumont) Wed, 08 Jun 2016 15:54:59 +0200 swh-model (0.0.9-1~swh1) unstable-swh; urgency=medium * v0.0.9 * Add coverage on edge case * Optimize git hash walk -- Antoine R. Dumont (@ardumont) Thu, 26 May 2016 12:56:17 +0200 swh-model (0.0.8-1~swh1) unstable-swh; urgency=medium * v0.0.8 * Add coverage on edge case * Optimize git hash walk -- Antoine R. Dumont (@ardumont) Thu, 26 May 2016 12:33:59 +0200 swh-model (0.0.7-1~swh1) unstable-swh; urgency=medium * v0.0.7 * Improve corner case policy about walking and computing hash tree (+ update) -- Antoine R. Dumont (@ardumont) Wed, 25 May 2016 23:47:19 +0200 swh-model (0.0.6-1~swh1) unstable-swh; urgency=medium * v0.0.6 * Improve corner case on git hash memory update function * debian packaging: Ignore fs tests for packaging -- Antoine R. Dumont (@ardumont) Tue, 24 May 2016 17:01:06 +0200 swh-model (0.0.5-1~swh1) unstable-swh; urgency=medium * v0.0.5 * Add update git hash computation from existing data * Add revision identifier data for hash identifier computation (extra- headers) -- Antoine R. Dumont (@ardumont) Fri, 15 Apr 2016 12:51:21 +0200 swh-model (0.0.4-1~swh1) unstable-swh; urgency=medium * v0.0.4 * Migrate swh.loader.dir.git module to swh.model.git -- Antoine R. Dumont (@ardumont) Mon, 21 Mar 2016 15:20:28 +0100 swh-model (0.0.3-1~swh1) unstable-swh; urgency=medium * v0.0.3 * Release name is now in bytes -- Antoine R. Dumont (@ardumont) Wed, 27 Jan 2016 15:50:08 +0100 swh-model (0.0.2-1~swh1) unstable-swh; urgency=medium * Prepare release of v0.0.2 * Import the rest of swh.core.hashutil -- Nicolas Dandrimont Wed, 16 Dec 2015 18:30:12 +0100 swh-model (0.0.1-1~swh1) unstable-swh; urgency=medium * Initial release * Prepare swh.model release v0.0.1 -- Nicolas Dandrimont Mon, 07 Dec 2015 18:26:58 +0100 diff --git a/swh.model.egg-info/PKG-INFO b/swh.model.egg-info/PKG-INFO index e14129d..92d127e 100644 --- a/swh.model.egg-info/PKG-INFO +++ b/swh.model.egg-info/PKG-INFO @@ -1,42 +1,42 @@ Metadata-Version: 2.1 Name: swh.model -Version: 6.5.0 +Version: 6.5.1 Summary: Software Heritage data model Home-page: https://forge.softwareheritage.org/diffusion/DMOD/ Author: Software Heritage developers Author-email: swh-devel@inria.fr Project-URL: Bug Reports, https://forge.softwareheritage.org/maniphest Project-URL: Funding, https://www.softwareheritage.org/donate Project-URL: Source, https://forge.softwareheritage.org/source/swh-model Project-URL: Documentation, https://docs.softwareheritage.org/devel/swh-model/ Classifier: Programming Language :: Python :: 3 Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3) Classifier: Operating System :: OS Independent Classifier: Development Status :: 5 - Production/Stable Requires-Python: >=3.7 Description-Content-Type: text/markdown Provides-Extra: cli Provides-Extra: testing-minimal Provides-Extra: testing License-File: LICENSE License-File: AUTHORS swh-model ========= Implementation of the Data model of the Software Heritage project, used to archive source code artifacts. This module defines the notion of SoftWare Heritage persistent IDentifiers (SWHIDs) and provides tools to compute them: ```sh $ swh-identify fork.c kmod.c sched/deadline.c swh:1:cnt:2e391c754ae730bd2d8520c2ab497c403220c6e3 fork.c swh:1:cnt:0277d1216f80ae1adeed84a686ed34c9b2931fc2 kmod.c swh:1:cnt:57b939c81bce5d06fa587df8915f05affbe22b82 sched/deadline.c $ swh-identify --no-filename /usr/src/linux/kernel/ swh:1:dir:f9f858a48d663b3809c9e2f336412717496202ab ``` diff --git a/swh/model/from_disk.py b/swh/model/from_disk.py index 86ecf12..8795b1f 100644 --- a/swh/model/from_disk.py +++ b/swh/model/from_disk.py @@ -1,588 +1,592 @@ # Copyright (C) 2017-2020 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU General Public License version 3, or any later version # See top-level LICENSE file for more information """Conversion from filesystem tree to SWH objects. This module allows reading a tree of directories and files from a local filesystem, and convert them to in-memory data structures, which can then be exported to SWH data model objects, as defined in :mod:`swh.model.model`. """ import datetime import enum import fnmatch import glob import os import re import stat from typing import Any, Iterable, Iterator, List, Optional, Pattern, Tuple import attr from attrs_strict import type_validator from typing_extensions import Final from . import model from .exceptions import InvalidDirectoryPath from .git_objects import directory_entry_sort_key from .hashutil import MultiHash, hash_to_hex from .merkle import MerkleLeaf, MerkleNode from .swhids import CoreSWHID, ObjectType @attr.s(frozen=True, slots=True) class DiskBackedContent(model.BaseContent): """Content-like class, which allows lazy-loading data from the disk.""" object_type: Final = "content_file" sha1 = attr.ib(type=bytes, validator=type_validator()) sha1_git = attr.ib(type=model.Sha1Git, validator=type_validator()) sha256 = attr.ib(type=bytes, validator=type_validator()) blake2s256 = attr.ib(type=bytes, validator=type_validator()) length = attr.ib(type=int, validator=type_validator()) status = attr.ib( type=str, validator=attr.validators.in_(["visible", "hidden"]), default="visible", ) ctime = attr.ib( type=Optional[datetime.datetime], validator=type_validator(), default=None, eq=False, ) path = attr.ib(type=Optional[bytes], default=None) @classmethod def from_dict(cls, d): return cls(**d) def __attrs_post_init__(self): if self.path is None: raise TypeError("path must not be None.") def with_data(self) -> model.Content: args = self.to_dict() del args["path"] assert self.path is not None with open(self.path, "rb") as fd: return model.Content.from_dict({**args, "data": fd.read()}) class DentryPerms(enum.IntEnum): """Admissible permissions for directory entries.""" content = 0o100644 """Content""" executable_content = 0o100755 """Executable content (e.g. executable script)""" symlink = 0o120000 """Symbolic link""" directory = 0o040000 """Directory""" revision = 0o160000 """Revision (e.g. submodule)""" def mode_to_perms(mode): """Convert a file mode to a permission compatible with Software Heritage directory entries Args: mode (int): a file mode as returned by :func:`os.stat` in :attr:`os.stat_result.st_mode` Returns: DentryPerms: one of the following values: :const:`DentryPerms.content`: plain file :const:`DentryPerms.executable_content`: executable file :const:`DentryPerms.symlink`: symbolic link :const:`DentryPerms.directory`: directory """ if stat.S_ISLNK(mode): return DentryPerms.symlink if stat.S_ISDIR(mode): return DentryPerms.directory else: # file is executable in any way if mode & (0o111): return DentryPerms.executable_content else: return DentryPerms.content class Content(MerkleLeaf): """Representation of a Software Heritage content as a node in a Merkle tree. The current Merkle hash for the Content nodes is the `sha1_git`, which makes it consistent with what :class:`Directory` uses for its own hash computation. """ __slots__ = [] # type: List[str] object_type: Final = "content" @classmethod def from_bytes(cls, *, mode, data): """Convert data (raw :class:`bytes`) to a Software Heritage content entry Args: mode (int): a file mode (passed to :func:`mode_to_perms`) data (bytes): raw contents of the file """ ret = MultiHash.from_data(data).digest() ret["length"] = len(data) ret["perms"] = mode_to_perms(mode) ret["data"] = data ret["status"] = "visible" return cls(ret) @classmethod def from_symlink(cls, *, path, mode): """Convert a symbolic link to a Software Heritage content entry""" return cls.from_bytes(mode=mode, data=os.readlink(path)) @classmethod def from_file(cls, *, path, max_content_length=None): """Compute the Software Heritage content entry corresponding to an on-disk file. The returned dictionary contains keys useful for both: - loading the content in the archive (hashes, `length`) - using the content as a directory entry in a directory Args: save_path (bool): add the file path to the entry max_content_length (Optional[int]): if given, all contents larger than this will be skipped. """ file_stat = os.lstat(path) mode = file_stat.st_mode length = file_stat.st_size too_large = max_content_length is not None and length > max_content_length if stat.S_ISLNK(mode): # Symbolic link: return a file whose contents are the link target if too_large: # Unlike large contents, we can't stream symlinks to # MultiHash, and we don't want to fit them in memory if # they exceed max_content_length either. # Thankfully, this should not happen for reasonable values of # max_content_length because of OS/filesystem limitations, # so let's just raise an error. raise Exception(f"Symlink too large ({length} bytes)") return cls.from_symlink(path=path, mode=mode) elif not stat.S_ISREG(mode): # not a regular file: return the empty file instead return cls.from_bytes(mode=mode, data=b"") if too_large: skip_reason = "Content too large" else: skip_reason = None hashes = MultiHash.from_path(path).digest() if skip_reason: ret = { **hashes, "status": "absent", "reason": skip_reason, } else: ret = { **hashes, "status": "visible", } ret["path"] = path ret["perms"] = mode_to_perms(mode) ret["length"] = length obj = cls(ret) return obj def swhid(self) -> CoreSWHID: """Return node identifier as a SWHID""" return CoreSWHID(object_type=ObjectType.CONTENT, object_id=self.hash) def __repr__(self): return "Content(id=%s)" % hash_to_hex(self.hash) def compute_hash(self): return self.data["sha1_git"] def to_model(self) -> model.BaseContent: """Builds a `model.BaseContent` object based on this leaf.""" data = self.get_data().copy() data.pop("perms", None) if data["status"] == "absent": data.pop("path", None) return model.SkippedContent.from_dict(data) elif "data" in data: return model.Content.from_dict(data) else: return DiskBackedContent.from_dict(data) def accept_all_directories(dirpath: str, dirname: str, entries: Iterable[Any]) -> bool: """Default filter for :func:`Directory.from_disk` accepting all directories Args: dirname (bytes): directory name entries (list): directory entries """ return True def ignore_empty_directories( dirpath: str, dirname: str, entries: Iterable[Any] ) -> bool: """Filter for :func:`directory_to_objects` ignoring empty directories Args: dirname (bytes): directory name entries (list): directory entries Returns: True if the directory is not empty, false if the directory is empty """ return bool(entries) def ignore_named_directories(names, *, case_sensitive=True): """Filter for :func:`directory_to_objects` to ignore directories named one of names. Args: names (list of bytes): names to ignore case_sensitive (bool): whether to do the filtering in a case sensitive way Returns: a directory filter for :func:`directory_to_objects` """ if not case_sensitive: names = [name.lower() for name in names] def named_filter( dirpath: str, dirname: str, entries: Iterable[Any], names: Iterable[Any] = names, case_sensitive: bool = case_sensitive, ): if case_sensitive: return dirname not in names else: return dirname.lower() not in names return named_filter # TODO: `extract_regex_objs` has been copied and adapted from `swh.scanner`. # In the future `swh.scanner` should use the `swh.model` version and remove its own. def extract_regex_objs( root_path: bytes, patterns: Iterable[bytes] ) -> Iterator[Pattern[bytes]]: """Generates a regex object for each pattern given in input and checks if the path is a subdirectory or relative to the root path. Args: root_path (bytes): path to the root directory - patterns (list of byte): patterns to match + patterns (list of byte): shell patterns to match Yields: an SRE_Pattern object """ absolute_root_path = os.path.abspath(root_path) for pattern in patterns: - for path in glob.glob(pattern): - absolute_path = os.path.abspath(path) - if not absolute_path.startswith(absolute_root_path): + if os.path.isabs(pattern): + pattern = os.path.relpath(pattern, root_path) + # python 3.10 has a `root_dir` argument for glob, but not the previous + # version. So we adjust the pattern + test_pattern = os.path.join(absolute_root_path, pattern) + for path in glob.glob(test_pattern): + if os.path.isabs(path) and not path.startswith(absolute_root_path): error_msg = ( b'The path "' + path + b'" is not a subdirectory or relative ' b'to the root directory path: "' + root_path + b'"' ) raise InvalidDirectoryPath(error_msg) regex = fnmatch.translate((pattern.decode())) yield re.compile(regex.encode()) def ignore_directories_patterns(root_path: bytes, patterns: Iterable[bytes]): """Filter for :func:`directory_to_objects` to ignore directories matching certain patterns. Args: root_path (bytes): path of the root directory - patterns (list of byte): patterns to ignore + patterns (list of bytes): patterns to ignore Returns: a directory filter for :func:`directory_to_objects` """ sre_patterns = set(extract_regex_objs(root_path, patterns)) def pattern_filter( dirpath: bytes, dirname: bytes, entries: Iterable[Any], patterns: Iterable[Any] = sre_patterns, root_path: bytes = os.path.abspath(root_path), ): full_path = os.path.abspath(dirpath) relative_path = os.path.relpath(full_path, root_path) return not any([pattern.match(relative_path) for pattern in patterns]) return pattern_filter def iter_directory( directory, ) -> Tuple[List[model.Content], List[model.SkippedContent], List[model.Directory]]: """Return the directory listing from a disk-memory directory instance. Raises: TypeError in case an unexpected object type is listed. Returns: Tuple of respectively iterable of content, skipped content and directories. """ contents: List[model.Content] = [] skipped_contents: List[model.SkippedContent] = [] directories: List[model.Directory] = [] for obj in directory.iter_tree(): obj = obj.to_model() obj_type = obj.object_type if obj_type in (model.Content.object_type, DiskBackedContent.object_type): # FIXME: read the data from disk later (when the # storage buffer is flushed). obj = obj.with_data() contents.append(obj) elif obj_type == model.SkippedContent.object_type: skipped_contents.append(obj) elif obj_type == model.Directory.object_type: directories.append(obj) else: raise TypeError(f"Unexpected object type from disk: {obj}") return contents, skipped_contents, directories class Directory(MerkleNode): """Representation of a Software Heritage directory as a node in a Merkle Tree. This class can be used to generate, from an on-disk directory, all the objects that need to be sent to the Software Heritage archive. The :func:`from_disk` constructor allows you to generate the data structure from a directory on disk. The resulting :class:`Directory` can then be manipulated as a dictionary, using the path as key. The :func:`collect` method is used to retrieve all the objects that need to be added to the Software Heritage archive since the last collection, by class (contents and directories). When using the dict-like methods to update the contents of the directory, the affected levels of hierarchy are reset and can be collected again using the same method. This enables the efficient collection of updated nodes, for instance when the client is applying diffs. """ __slots__ = ["__entries", "__model_object"] object_type: Final = "directory" @classmethod def from_disk( cls, *, path, dir_filter=accept_all_directories, max_content_length=None ): """Compute the Software Heritage objects for a given directory tree Args: path (bytes): the directory to traverse data (bool): whether to add the data to the content objects save_path (bool): whether to add the path to the content objects dir_filter (function): a filter to ignore some directories by name or contents. Takes two arguments: dirname and entries, and returns True if the directory should be added, False if the directory should be ignored. max_content_length (Optional[int]): if given, all contents larger than this will be skipped. """ top_path = path dirs = {} for root, dentries, fentries in os.walk(top_path, topdown=False): entries = {} # Join fentries and dentries in the same processing, as symbolic # links to directories appear in dentries... for name in fentries + dentries: path = os.path.join(root, name) if not os.path.isdir(path) or os.path.islink(path): content = Content.from_file( path=path, max_content_length=max_content_length ) entries[name] = content else: if dir_filter(path, name, dirs[path].entries): entries[name] = dirs[path] dirs[root] = cls({"name": os.path.basename(root), "path": root}) dirs[root].update(entries) return dirs[top_path] def __init__(self, data=None): super().__init__(data=data) self.__entries = None self.__model_object = None def invalidate_hash(self): self.__entries = None self.__model_object = None super().invalidate_hash() @staticmethod def child_to_directory_entry(name, child): if child.object_type == "directory": return { "type": "dir", "perms": DentryPerms.directory, "target": child.hash, "name": name, } elif child.object_type == "content": return { "type": "file", "perms": child.data["perms"], "target": child.hash, "name": name, } else: raise ValueError(f"unknown child {child}") def get_data(self, **kwargs): return { "id": self.hash, "entries": self.entries, } @property def entries(self): """Child nodes, sorted by name in the same way :func:`swh.model.git_objects.directory_git_object` does.""" if self.__entries is None: self.__entries = sorted( ( self.child_to_directory_entry(name, child) for name, child in self.items() ), key=directory_entry_sort_key, ) return self.__entries def swhid(self) -> CoreSWHID: """Return node identifier as a SWHID""" return CoreSWHID(object_type=ObjectType.DIRECTORY, object_id=self.hash) def compute_hash(self): return self.to_model().id def to_model(self) -> model.Directory: """Builds a `model.Directory` object based on this node; ignoring its children.""" if self.__model_object is None: DirectoryEntry = model.DirectoryEntry entries = [] for name, child in self.items(): if child.object_type == "directory": e = DirectoryEntry( type="dir", perms=DentryPerms.directory, target=child.hash, name=name, ) elif child.object_type == "content": e = DirectoryEntry( type="file", perms=child.data["perms"], target=child.hash, name=name, ) else: raise ValueError(f"unknown child {child}") entries.append(e) entries.sort(key=directory_entry_sort_key) self.__model_object = model.Directory(entries=tuple(entries)) return self.__model_object def __getitem__(self, key): if not isinstance(key, bytes): raise ValueError("Can only get a bytes from Directory") # Convenience shortcut if key == b"": return self if b"/" not in key: return super().__getitem__(key) else: key1, key2 = key.split(b"/", 1) return self.__getitem__(key1)[key2] def __setitem__(self, key, value): if not isinstance(key, bytes): raise ValueError("Can only set a bytes Directory entry") if not isinstance(value, (Content, Directory)): raise ValueError( "Can only set a Directory entry to a Content or " "Directory" ) if key == b"": raise ValueError("Directory entry must have a name") if b"\x00" in key: raise ValueError("Directory entry name must not contain nul bytes") if b"/" not in key: return super().__setitem__(key, value) else: key1, key2 = key.rsplit(b"/", 1) self[key1].__setitem__(key2, value) def __delitem__(self, key): if not isinstance(key, bytes): raise ValueError("Can only delete a bytes Directory entry") if b"/" not in key: super().__delitem__(key) else: key1, key2 = key.rsplit(b"/", 1) del self[key1][key2] def __contains__(self, key): if b"/" not in key: return super().__contains__(key) else: key1, key2 = key.split(b"/", 1) return super().__contains__(key1) and self[key1].__contains__(key2) def __repr__(self): return "Directory(id=%s, entries=[%s])" % ( hash_to_hex(self.hash), ", ".join(str(entry) for entry in self), ) diff --git a/swh/model/hashutil.py b/swh/model/hashutil.py index 75d9f8b..8d2cb90 100644 --- a/swh/model/hashutil.py +++ b/swh/model/hashutil.py @@ -1,351 +1,353 @@ # Copyright (C) 2015-2018 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU General Public License version 3, or any later version # See top-level LICENSE file for more information """Module in charge of hashing function definitions. This is the base module use to compute swh's hashes. Only a subset of hashing algorithms is supported as defined in the ALGORITHMS set. Any provided algorithms not in that list will result in a ValueError explaining the error. This module defines a MultiHash class to ease the softwareheritage hashing algorithms computation. This allows to compute hashes from file object, path, data using a similar interface as what the standard hashlib module provides. Basic usage examples: - file object: MultiHash.from_file( file_object, hash_names=DEFAULT_ALGORITHMS).digest() - path (filepath): MultiHash.from_path(b'foo').hexdigest() - data (bytes): MultiHash.from_data(b'foo').bytehexdigest() "Complex" usage, defining a swh hashlib instance first: - To compute length, integrate the length to the set of algorithms to compute, for example: .. code-block:: python h = MultiHash(hash_names=set({'length'}).union(DEFAULT_ALGORITHMS)) with open(filepath, 'rb') as f: h.update(f.read(HASH_BLOCK_SIZE)) hashes = h.digest() # returns a dict of {hash_algo_name: hash_in_bytes} - Write alongside computing hashing algorithms (from a stream), example: .. code-block:: python h = MultiHash(length=length) with open(filepath, 'wb') as f: for chunk in r.iter_content(): # r a stream of sort h.update(chunk) f.write(chunk) hashes = h.hexdigest() # returns a dict of {hash_algo_name: hash_in_hex} """ import binascii import functools import hashlib from io import BytesIO import os from typing import Callable, Dict, Optional, Union -ALGORITHMS = set(["sha1", "sha256", "sha1_git", "blake2s256", "blake2b512", "md5"]) +ALGORITHMS = set( + ["sha1", "sha256", "sha1_git", "blake2s256", "blake2b512", "md5", "sha512"] +) """Hashing algorithms supported by this module""" DEFAULT_ALGORITHMS = set(["sha1", "sha256", "sha1_git", "blake2s256"]) """Algorithms computed by default when calling the functions from this module. Subset of :const:`ALGORITHMS`. """ HASH_BLOCK_SIZE = 32768 """Block size for streaming hash computations made in this module""" _blake2_hash_cache = {} # type: Dict[str, Callable] class MultiHash: """Hashutil class to support multiple hashes computation. Args: hash_names (set): Set of hash algorithms (+ optionally length) to compute hashes (cf. DEFAULT_ALGORITHMS) length (int): Length of the total sum of chunks to read If the length is provided as algorithm, the length is also computed and returned. """ def __init__(self, hash_names=DEFAULT_ALGORITHMS, length=None): self.state = {} self.track_length = False for name in hash_names: if name == "length": self.state["length"] = 0 self.track_length = True else: self.state[name] = _new_hash(name, length) @classmethod def from_state(cls, state, track_length): ret = cls([]) ret.state = state ret.track_length = track_length @classmethod def from_file(cls, fobj, hash_names=DEFAULT_ALGORITHMS, length=None): ret = cls(length=length, hash_names=hash_names) while True: chunk = fobj.read(HASH_BLOCK_SIZE) if not chunk: break ret.update(chunk) return ret @classmethod def from_path(cls, path, hash_names=DEFAULT_ALGORITHMS): length = os.path.getsize(path) with open(path, "rb") as f: ret = cls.from_file(f, hash_names=hash_names, length=length) return ret @classmethod def from_data(cls, data, hash_names=DEFAULT_ALGORITHMS): length = len(data) fobj = BytesIO(data) return cls.from_file(fobj, hash_names=hash_names, length=length) def update(self, chunk): for name, h in self.state.items(): if name == "length": continue h.update(chunk) if self.track_length: self.state["length"] += len(chunk) def digest(self): return { name: h.digest() if name != "length" else h for name, h in self.state.items() } def hexdigest(self): return { name: h.hexdigest() if name != "length" else h for name, h in self.state.items() } def bytehexdigest(self): return { name: hash_to_bytehex(h.digest()) if name != "length" else h for name, h in self.state.items() } def copy(self): copied_state = { name: h.copy() if name != "length" else h for name, h in self.state.items() } return self.from_state(copied_state, self.track_length) def _new_blake2_hash(algo): """Return a function that initializes a blake2 hash.""" if algo in _blake2_hash_cache: return _blake2_hash_cache[algo]() lalgo = algo.lower() if not lalgo.startswith("blake2"): raise ValueError("Algorithm %s is not a blake2 hash" % algo) blake_family = lalgo[:7] digest_size = None if lalgo[7:]: try: digest_size, remainder = divmod(int(lalgo[7:]), 8) except ValueError: raise ValueError("Unknown digest size for algo %s" % algo) from None if remainder: raise ValueError( "Digest size for algorithm %s must be a multiple of 8" % algo ) blake2 = getattr(hashlib, blake_family) _blake2_hash_cache[algo] = lambda: blake2(digest_size=digest_size) return _blake2_hash_cache[algo]() def _new_hashlib_hash(algo): """Initialize a digest object from hashlib. Handle the swh-specific names for the blake2-related algorithms """ if algo.startswith("blake2"): return _new_blake2_hash(algo) else: return hashlib.new(algo) def git_object_header(git_type: str, length: int) -> bytes: """Returns the header for a git object of the given type and length. The header of a git object consists of: - The type of the object (encoded in ASCII) - One ASCII space (\x20) - The length of the object (decimal encoded in ASCII) - One NUL byte Args: base_algo (str from :const:`ALGORITHMS`): a hashlib-supported algorithm git_type: the type of the git object (supposedly one of 'blob', 'commit', 'tag', 'tree') length: the length of the git object you're encoding Returns: a hashutil.hash object """ git_object_types = { "blob", "tree", "commit", "tag", "snapshot", "raw_extrinsic_metadata", "extid", } if git_type not in git_object_types: raise ValueError( "Unexpected git object type %s, expected one of %s" % (git_type, ", ".join(sorted(git_object_types))) ) return ("%s %d\0" % (git_type, length)).encode("ascii") def _new_hash(algo: str, length: Optional[int] = None): """Initialize a digest object (as returned by python's hashlib) for the requested algorithm. See the constant ALGORITHMS for the list of supported algorithms. If a git-specific hashing algorithm is requested (e.g., "sha1_git"), the hashing object will be pre-fed with the needed header; for this to work, length must be given. Args: algo (str): a hashing algorithm (one of ALGORITHMS) length (int): the length of the hashed payload (needed for git-specific algorithms) Returns: a hashutil.hash object Raises: ValueError if algo is unknown, or length is missing for a git-specific hash. """ if algo not in ALGORITHMS: raise ValueError( "Unexpected hashing algorithm %s, expected one of %s" % (algo, ", ".join(sorted(ALGORITHMS))) ) if algo.endswith("_git"): if length is None: raise ValueError("Missing length for git hashing algorithm") base_algo = algo[:-4] h = _new_hashlib_hash(base_algo) h.update(git_object_header("blob", length)) return h return _new_hashlib_hash(algo) def hash_git_data(data, git_type, base_algo="sha1"): """Hash the given data as a git object of type git_type. Args: data: a bytes object git_type: the git object type base_algo: the base hashing algorithm used (default: sha1) Returns: a dict mapping each algorithm to a bytes digest Raises: ValueError if the git_type is unexpected. """ h = _new_hashlib_hash(base_algo) h.update(git_object_header(git_type, len(data))) h.update(data) return h.digest() @functools.lru_cache() def hash_to_hex(hash: Union[str, bytes]) -> str: """Converts a hash (in hex or bytes form) to its hexadecimal ascii form Args: hash (str or bytes): a :class:`bytes` hash or a :class:`str` containing the hexadecimal form of the hash Returns: str: the hexadecimal form of the hash """ if isinstance(hash, str): return hash return binascii.hexlify(hash).decode("ascii") @functools.lru_cache() def hash_to_bytehex(hash: bytes) -> bytes: """Converts a hash to its hexadecimal bytes representation Args: hash (bytes): a :class:`bytes` hash Returns: bytes: the hexadecimal form of the hash, as :class:`bytes` """ return binascii.hexlify(hash) @functools.lru_cache() def hash_to_bytes(hash: Union[str, bytes]) -> bytes: """Converts a hash (in hex or bytes form) to its raw bytes form Args: hash (str or bytes): a :class:`bytes` hash or a :class:`str` containing the hexadecimal form of the hash Returns: bytes: the :class:`bytes` form of the hash """ if isinstance(hash, bytes): return hash return bytes.fromhex(hash) @functools.lru_cache() def bytehex_to_hash(hex: bytes) -> bytes: """Converts a hexadecimal bytes representation of a hash to that hash Args: hash (bytes): a :class:`bytes` containing the hexadecimal form of the hash encoded in ascii Returns: bytes: the :class:`bytes` form of the hash """ return hash_to_bytes(hex.decode()) diff --git a/swh/model/tests/test_hashutil.py b/swh/model/tests/test_hashutil.py index 1ab2812..b279f2f 100644 --- a/swh/model/tests/test_hashutil.py +++ b/swh/model/tests/test_hashutil.py @@ -1,324 +1,334 @@ # Copyright (C) 2015-2021 The Software Heritage developers # See the AUTHORS file at the top-level directory of this distribution # License: GNU General Public License version 3, or any later version # See top-level LICENSE file for more information import contextlib import hashlib import io import os import tempfile from unittest.mock import patch import pytest from swh.model import hashutil from swh.model.hashutil import DEFAULT_ALGORITHMS, MultiHash, hash_to_bytehex @contextlib.contextmanager def patch_blake2(function_name): try: with patch(function_name) as mock: yield mock finally: # mocking blake2 inserts mock objects in the cache; we need # to clean it before the next test runs hashutil._blake2_hash_cache.clear() @pytest.fixture(autouse=True) def blake2_hash_cache_reset(): # Reset function cache hashutil._blake2_hash_cache = {} @pytest.fixture def hash_test_data(): class HashTestData: data = b"1984\n" hex_checksums = { "sha1": "62be35bf00ff0c624f4a621e2ea5595a049e0731", "sha1_git": "568aaf43d83b2c3df8067f3bedbb97d83260be6d", "sha256": "26602113b4b9afd9d55466b08580d3c2" "4a9b50ee5b5866c0d91fab0e65907311", "blake2s256": "63cfb259e1fdb485bc5c55749697a6b21ef31fb7445f6c78a" "c9422f9f2dc8906", } checksums = { type: bytes.fromhex(cksum) for type, cksum in hex_checksums.items() } bytehex_checksums = { type: hashutil.hash_to_bytehex(cksum) for type, cksum in checksums.items() } git_hex_checksums = { "blob": hex_checksums["sha1_git"], "tree": "5b2e883aa33d2efab98442693ea4dd5f1b8871b0", "commit": "79e4093542e72f0fcb7cbd75cb7d270f9254aa8f", "tag": "d6bf62466f287b4d986c545890716ce058bddf67", } git_checksums = { type: bytes.fromhex(cksum) for type, cksum in git_hex_checksums.items() } return HashTestData def test_multi_hash_data(hash_test_data): checksums = MultiHash.from_data(hash_test_data.data).digest() assert checksums == hash_test_data.checksums assert "length" not in checksums def test_multi_hash_data_with_length(hash_test_data): expected_checksums = hash_test_data.checksums.copy() expected_checksums["length"] = len(hash_test_data.data) algos = set(["length"]).union(hashutil.DEFAULT_ALGORITHMS) checksums = MultiHash.from_data(hash_test_data.data, hash_names=algos).digest() assert checksums == expected_checksums assert "length" in checksums def test_multi_hash_data_unknown_hash(hash_test_data): with pytest.raises(ValueError, match="Unexpected hashing algorithm.*unknown-hash"): MultiHash.from_data(hash_test_data.data, ["unknown-hash"]) def test_multi_hash_file(hash_test_data): fobj = io.BytesIO(hash_test_data.data) checksums = MultiHash.from_file(fobj, length=len(hash_test_data.data)).digest() assert checksums == hash_test_data.checksums def test_multi_hash_file_hexdigest(hash_test_data): fobj = io.BytesIO(hash_test_data.data) length = len(hash_test_data.data) checksums = MultiHash.from_file(fobj, length=length).hexdigest() assert checksums == hash_test_data.hex_checksums def test_multi_hash_file_bytehexdigest(hash_test_data): fobj = io.BytesIO(hash_test_data.data) length = len(hash_test_data.data) checksums = MultiHash.from_file(fobj, length=length).bytehexdigest() assert checksums == hash_test_data.bytehex_checksums -def test_multi_hash_file_with_md5(hash_test_data): +EXTRA_HASH_ALGOS = ["md5", "sha512"] + + +@pytest.mark.parametrize("hash_algo", EXTRA_HASH_ALGOS) +def test_multi_hash_file_with_extra_hash_algo(hash_test_data, hash_algo): fobj = io.BytesIO(hash_test_data.data) checksums = MultiHash.from_file( - fobj, hash_names=DEFAULT_ALGORITHMS | {"md5"}, length=len(hash_test_data.data) + fobj, + hash_names=DEFAULT_ALGORITHMS | {hash_algo}, + length=len(hash_test_data.data), ).digest() - md5sum = {"md5": hashlib.md5(hash_test_data.data).digest()} - assert checksums == {**hash_test_data.checksums, **md5sum} + checksum = {hash_algo: hashlib.new(hash_algo, hash_test_data.data).digest()} + assert checksums == {**hash_test_data.checksums, **checksum} -def test_multi_hash_file_hexdigest_with_md5(hash_test_data): +@pytest.mark.parametrize("hash_algo", EXTRA_HASH_ALGOS) +def test_multi_hash_file_hexdigest_with_extra_hash_algo(hash_test_data, hash_algo): fobj = io.BytesIO(hash_test_data.data) length = len(hash_test_data.data) checksums = MultiHash.from_file( - fobj, hash_names=DEFAULT_ALGORITHMS | {"md5"}, length=length + fobj, hash_names=DEFAULT_ALGORITHMS | {hash_algo}, length=length ).hexdigest() - md5sum = {"md5": hashlib.md5(hash_test_data.data).hexdigest()} - assert checksums == {**hash_test_data.hex_checksums, **md5sum} + checksum = {hash_algo: hashlib.new(hash_algo, hash_test_data.data).hexdigest()} + assert checksums == {**hash_test_data.hex_checksums, **checksum} -def test_multi_hash_file_bytehexdigest_with_md5(hash_test_data): +@pytest.mark.parametrize("hash_algo", EXTRA_HASH_ALGOS) +def test_multi_hash_file_bytehexdigest_with_extra_algo(hash_test_data, hash_algo): fobj = io.BytesIO(hash_test_data.data) length = len(hash_test_data.data) checksums = MultiHash.from_file( - fobj, hash_names=DEFAULT_ALGORITHMS | {"md5"}, length=length + fobj, hash_names=DEFAULT_ALGORITHMS | {hash_algo}, length=length ).bytehexdigest() - md5sum = {"md5": hash_to_bytehex(hashlib.md5(hash_test_data.data).digest())} - assert checksums == {**hash_test_data.bytehex_checksums, **md5sum} + checksum = { + hash_algo: hash_to_bytehex(hashlib.new(hash_algo, hash_test_data.data).digest()) + } + assert checksums == {**hash_test_data.bytehex_checksums, **checksum} def test_multi_hash_file_missing_length(hash_test_data): fobj = io.BytesIO(hash_test_data.data) with pytest.raises(ValueError, match="Missing length"): MultiHash.from_file(fobj, hash_names=["sha1_git"]) def test_multi_hash_path(hash_test_data): with tempfile.NamedTemporaryFile(delete=False) as f: f.write(hash_test_data.data) hashes = MultiHash.from_path(f.name).digest() os.remove(f.name) assert hash_test_data.checksums == hashes def test_hash_git_data(hash_test_data): checksums = { git_type: hashutil.hash_git_data(hash_test_data.data, git_type) for git_type in hash_test_data.git_checksums } assert checksums == hash_test_data.git_checksums def test_hash_git_data_unknown_git_type(hash_test_data): with pytest.raises( ValueError, match="Unexpected git object type.*unknown-git-type" ): hashutil.hash_git_data(hash_test_data.data, "unknown-git-type") def test_hash_to_hex(hash_test_data): for type in hash_test_data.checksums: hex = hash_test_data.hex_checksums[type] hash = hash_test_data.checksums[type] assert hashutil.hash_to_hex(hex) == hex assert hashutil.hash_to_hex(hash) == hex def test_hash_to_bytes(hash_test_data): for type in hash_test_data.checksums: hex = hash_test_data.hex_checksums[type] hash = hash_test_data.checksums[type] assert hashutil.hash_to_bytes(hex) == hash assert hashutil.hash_to_bytes(hash) == hash def test_hash_to_bytehex(hash_test_data): for algo in hash_test_data.checksums: hex_checksum = hash_test_data.hex_checksums[algo].encode("ascii") assert hex_checksum == hashutil.hash_to_bytehex(hash_test_data.checksums[algo]) def test_bytehex_to_hash(hash_test_data): for algo in hash_test_data.checksums: assert hash_test_data.checksums[algo] == hashutil.bytehex_to_hash( hash_test_data.hex_checksums[algo].encode() ) def test_new_hash_unsupported_hashing_algorithm(): expected_message = ( "Unexpected hashing algorithm blake2:10, " "expected one of blake2b512, blake2s256, " "md5, sha1, sha1_git, sha256" ) with pytest.raises(ValueError, match=expected_message): hashutil._new_hash("blake2:10") def test_new_hash_blake2b_builtin(): with patch_blake2("hashlib.blake2b") as mock_blake2b: mock_blake2b.return_value = sentinel = object() h = hashutil._new_hash("blake2b512") assert h is sentinel mock_blake2b.assert_called_with(digest_size=512 // 8) def test_new_hash_blake2s_builtin(): with patch_blake2("hashlib.blake2s") as mock_blake2s: mock_blake2s.return_value = sentinel = object() h = hashutil._new_hash("blake2s256") assert h is sentinel mock_blake2s.assert_called_with(digest_size=256 // 8) @pytest.fixture def hashgit_test_data(): class HashGitTestData: blob_data = b"42\n" tree_data = b"".join( [ b"40000 barfoo\0", bytes.fromhex("c3020f6bf135a38c6df" "3afeb5fb38232c5e07087"), b"100644 blah\0", bytes.fromhex("63756ef0df5e4f10b6efa" "33cfe5c758749615f20"), b"100644 hello\0", bytes.fromhex("907b308167f0880fb2a" "5c0e1614bb0c7620f9dc3"), ] ) commit_data = b"""\ tree 1c61f7259dcb770f46b194d941df4f08ff0a3970 author Antoine R. Dumont (@ardumont) 1444054085 +0200 committer Antoine R. Dumont (@ardumont) 1444054085 +0200 initial """ # noqa tag_data = """object 24d012aaec0bc5a4d2f62c56399053d6cc72a241 type commit tag 0.0.1 tagger Antoine R. Dumont (@ardumont) 1444225145 +0200 blah """.encode( "utf-8" ) # NOQA checksums = { "blob_sha1_git": bytes.fromhex( "d81cc0710eb6cf9efd5b920a8453e1" "e07157b6cd" ), "tree_sha1_git": bytes.fromhex( "ac212302c45eada382b27bfda795db" "121dacdb1c" ), "commit_sha1_git": bytes.fromhex( "e960570b2e6e2798fa4cfb9af2c399" "d629189653" ), "tag_sha1_git": bytes.fromhex( "bc2b99ba469987bcf1272c189ed534" "e9e959f120" ), } return HashGitTestData def test_unknown_header_type(): with pytest.raises(ValueError, match="Unexpected git object type"): hashutil.hash_git_data(b"any-data", "some-unknown-type") def test_hashdata_content(hashgit_test_data): # when actual_hash = hashutil.hash_git_data(hashgit_test_data.blob_data, git_type="blob") # then assert actual_hash == hashgit_test_data.checksums["blob_sha1_git"] def test_hashdata_tree(hashgit_test_data): # when actual_hash = hashutil.hash_git_data(hashgit_test_data.tree_data, git_type="tree") # then assert actual_hash == hashgit_test_data.checksums["tree_sha1_git"] def test_hashdata_revision(hashgit_test_data): # when actual_hash = hashutil.hash_git_data( hashgit_test_data.commit_data, git_type="commit" ) # then assert actual_hash == hashgit_test_data.checksums["commit_sha1_git"] def test_hashdata_tag(hashgit_test_data): # when actual_hash = hashutil.hash_git_data(hashgit_test_data.tag_data, git_type="tag") # then assert actual_hash == hashgit_test_data.checksums["tag_sha1_git"]