replay: fix raw_extrinsic_metadata insertion and type annotation
ClosedPublic
Actions

Authored by douardda on Sep 28 2021, 5:18 PM.

Details

Reviewers

Group Reviewers

Commits

rDSTO113088ab065b: replay: add type annotation for process_replay_objects()
rDSTO21aff2d1cbab: replay: fix annotation of collision_aware_content_add()
rDSTO9a3589f24cae: replay: fix raw_extrinsic_metadata insertion and type annotation

Summary

due to missing type annotation of the storage argument of
_insert_objects(), we missed a bug in the processing of
raw_extrinsic_metadata objects, passing set() as arguments of storage
add methods.

Also fix annotation of collision_aware_content_add(),
the callable is expected to return a dict (and now we have an annotated storage,
mypy is not happy any more).

Diff Detail

Repository

rDSTO Storage manager

Branch

master

Lint

Lint Skipped

Unit

Unit Tests Skipped

Build Status

Buildable 24059
Build 37540: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 37539: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	3 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_revision 134 >>> pprint(rev1['metadata']['extra_headers']) 135 [['time_offset_seconds', b'-3600'], 136 ['transplant_source', b'29c154a012a70f49df983625090434587622b39e']]
	3 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_diff.TestDiffRevisions::test_insert_delete self = <swh.storage.tests.algos.test_diff.TestDiffRevisions testMethod=test_insert_delete> mock_get_dir = <MagicMock name='_get_dir' id='139985357210344'> mock_get_rev = <MagicMock name='_get_rev' id='139985357211352'>
	1 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_diff.TestDiffRevisions::test_insert_delete_empty_dirs self = <swh.storage.tests.algos.test_diff.TestDiffRevisions testMethod=test_insert_delete_empty_dirs> mock_get_dir = <MagicMock name='_get_dir' id='139985357289568'> mock_get_rev = <MagicMock name='_get_rev' id='139985354789384'>
	2 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_diff.TestDiffRevisions::test_onelevel_diff self = <swh.storage.tests.algos.test_diff.TestDiffRevisions testMethod=test_onelevel_diff> mock_get_dir = <MagicMock name='_get_dir' id='139985356699240'> mock_get_rev = <MagicMock name='_get_rev' id='139985386420544'>
	2 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.algos.test_diff.TestDiffRevisions::test_track_renaming self = <swh.storage.tests.algos.test_diff.TestDiffRevisions testMethod=test_track_renaming> mock_get_dir = <MagicMock name='_get_dir' id='139985386412184'> mock_get_rev = <MagicMock name='_get_rev' id='139986085857824'>
		View Full Test Results (9 Failed · 1,094 Passed · 40 Skipped)

Event Timeline

douardda created this revision.Sep 28 2021, 5:18 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptSep 28 2021, 5:18 PM

vlorentz published this revision for review.Sep 28 2021, 5:23 PM

vlorentz accepted this revision.

This revision is now accepted and ready to land.Sep 28 2021, 5:23 PM

Build has FAILED

Patch application report for D6370 (id=23146)

Rebasing onto 42bad90816...

Current branch diff-target is up to date.

Changes applied before test

commit 24dc0ee19a9b7dcc3229f7a2a4de565f78508144
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Sep 28 17:13:26 2021 +0200

    replay: fix raw_extrinsic_metadata insertion and type annotation
    
    due to missing type annotation of the storage argument of
    _insert_objects(), we missed a bug in the processing of
    raw_extrinsic_metadata objects, passing set() as arguments of storage
    add methods.

commit b963639542e8ad8f4deafc14433b79610608ebd9
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Sep 28 17:04:18 2021 +0200

    replay: fix annotation of collision_aware_content_add()
    
    now the callable is expected to return a dict.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1428/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1428/console

Harbormaster failed remote builds in B24059: Diff 23146!Sep 28 2021, 5:27 PM

rebase

Build is green

Patch application report for D6370 (id=23161)

Rebasing onto 8ba232c5ef...

Current branch diff-target is up to date.

Changes applied before test

commit 9a3589f24cae48cdd7cd73e512fcf92c5ec7cfa7
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Sep 28 17:13:26 2021 +0200

    replay: fix raw_extrinsic_metadata insertion and type annotation
    
    due to missing type annotation of the storage argument of
    _insert_objects(), we missed a bug in the processing of
    raw_extrinsic_metadata objects, passing set() as arguments of storage
    add methods.

commit 21aff2d1cbab92cc157e00cd466528a902c62a0f
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Sep 28 17:04:18 2021 +0200

    replay: fix annotation of collision_aware_content_add()
    
    now the callable is expected to return a dict.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1431/ for more details.

Harbormaster completed remote builds in B24076: Diff 23161.Sep 29 2021, 10:57 AM

add type annotation for process_replay_objects()

Build is green

Patch application report for D6370 (id=23163)

Rebasing onto 8ba232c5ef...

Current branch diff-target is up to date.

Changes applied before test

commit 113088ab065bcc7a089db92fd5e02ee89b1d0fe4
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Sep 29 10:59:12 2021 +0200

    replay: add type annotation for process_replay_objects()

commit 9a3589f24cae48cdd7cd73e512fcf92c5ec7cfa7
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Sep 28 17:13:26 2021 +0200

    replay: fix raw_extrinsic_metadata insertion and type annotation
    
    due to missing type annotation of the storage argument of
    _insert_objects(), we missed a bug in the processing of
    raw_extrinsic_metadata objects, passing set() as arguments of storage
    add methods.

commit 21aff2d1cbab92cc157e00cd466528a902c62a0f
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Sep 28 17:04:18 2021 +0200

    replay: fix annotation of collision_aware_content_add()
    
    now the callable is expected to return a dict.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1432/ for more details.

Harbormaster completed remote builds in B24078: Diff 23163.Sep 29 2021, 11:11 AM

Closed by commit rDSTO21aff2d1cbab: replay: fix annotation of collision_aware_content_add() (authored by douardda). · Explain WhySep 29 2021, 11:26 AM

This revision was automatically updated to reflect the committed changes.

douardda added a commit: rDSTO21aff2d1cbab: replay: fix annotation of collision_aware_content_add().

douardda added a commit: rDSTO9a3589f24cae: replay: fix raw_extrinsic_metadata insertion and type annotation.

douardda added a commit: rDSTO113088ab065b: replay: add type annotation for process_replay_objects().