Differential D3349

test_storage: Add missing tests on origin_visit_get method
ClosedPublic
Actions

Authored by ardumont on Jun 24 2020, 5:18 PM.

Details

Reviewers

vlorentz

Group Reviewers

Reviewers

Commits

rDSTOb991e697074a: test_storage: Add missing tests on origin_visit_get method

Summary

I'm planning to change the default order of the returned results (and test with
it) [1]

[1] D3350#82121 for more details.

Test Plan

tox

Diff Detail

Repository

rDSTO Storage manager

Branch

add-missing-test

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 13085
Build 19958: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 19957: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	3,105 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_backfill::test_backfiller swh_storage_backend_config = {'cls': 'local', 'db': 'postgresql://postgres@127.0.0.1:29727/tests', 'journal_writer': {'brokers': ['127.0.0.1:52229'], 'client_id': 'kafka_writer-1', 'cls': 'kafka', 'prefix': 'ghwzsjgfkj-1'}, 'objstorage': {'args': {}, 'cls': 'memory'}} kafka_prefix = 'ghwzsjgfkj', kafka_consumer_group = 'test-consumer-ghwzsjgfkj' kafka_server = '127.0.0.1:52229'
	3,625 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_kafka_writer::test_storage_direct_writer kafka_prefix = 'trwazqbiah', kafka_server = '127.0.0.1:52229' consumer = <cimpl.Consumer object at 0x7f7267ea8a60>
	5,006 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_replay::test_storage_play_with_collision replayer_storage_and_client = (<swh.storage.in_memory.InMemoryStorage object at 0x7f73e417ca58>, <swh.journal.client.JournalClient object at 0x7f73e417c4a8>) caplog = <_pytest.logging.LogCaptureFixture object at 0x7f732417ad68>
	4,005 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_replay::test_storage_replayer replayer_storage_and_client = (<swh.storage.in_memory.InMemoryStorage object at 0x7f726dca1da0>, <swh.journal.client.JournalClient object at 0x7f726dca5198>) caplog = <_pytest.logging.LogCaptureFixture object at 0x7f726c36df60>
	4 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_content
		View Full Test Results (4 Failed · 748 Passed · 17 Skipped)

Event Timeline

ardumont created this revision.Jun 24 2020, 5:18 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJun 24 2020, 5:18 PM

Build has FAILED

Patch application report for D3349 (id=11871)

Rebasing onto 621fc8d377...

Current branch diff-target is up to date.

Changes applied before test

commit e2981059d761a6773cbec4a327c5e72450834bbb
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 17:11:36 2020 +0200

    test_storage: Add missing tests on origin_visit_get method

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/344/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/344/console

Harbormaster failed remote builds in B13085: Diff 11871!Jun 24 2020, 5:24 PM

ardumont added a child revision: D3350: Iterate over paginated visits in batches to retrieve latest visit/snapshot.Jun 24 2020, 6:11 PM

Build has FAILED

Patch application report for D3349 (id=11871)

Rebasing onto 621fc8d377...

Current branch diff-target is up to date.

Changes applied before test

commit e2981059d761a6773cbec4a327c5e72450834bbb
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 17:11:36 2020 +0200

    test_storage: Add missing tests on origin_visit_get method

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/346/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/346/console

vlorentz accepted this revision.Jun 25 2020, 10:01 AM

vlorentz added a subscriber: vlorentz.

vlorentz added inline comments.

swh/storage/tests/test_storage.py
1460	nit: visits = swh_storage.origin_visit_add([visit1, visit2, visit3]) ov1, ov2, ov3 = [v.to_dict() for v in [ov1, ov2, ov3]] then you can get rid of the list comprehensions below.

This revision is now accepted and ready to land.Jun 25 2020, 10:01 AM

ardumont edited the summary of this revision. (Show Details)Jun 25 2020, 10:13 AM

ardumont edited the summary of this revision. (Show Details)

ardumont added inline comments.Jun 25 2020, 11:37 AM

swh/storage/tests/test_storage.py
1460	thanks ;)

Adapt according to review
Rebase on D3342 (so builds fine)

Build is green

Patch application report for D3349 (id=11888)

Could not rebase; Attempt merge onto 621fc8d377...

Updating 621fc8d3..7d7cb1dc
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/155.sql               |  34 +++
 swh/storage/backfill.py            |  10 +-
 swh/storage/cassandra/cql.py       |   3 -
 swh/storage/cassandra/schema.py    |   3 -
 swh/storage/cassandra/storage.py   |  15 +-
 swh/storage/db.py                  |  27 +--
 swh/storage/in_memory.py           |  18 +-
 swh/storage/sql/30-swh-schema.sql  |  11 +-
 swh/storage/sql/40-swh-func.sql    |   4 +-
 swh/storage/sql/60-swh-indexes.sql |   2 +-
 swh/storage/storage.py             |  10 +-
 swh/storage/tests/test_backfill.py |   5 +-
 swh/storage/tests/test_storage.py  | 455 +++++++++++++++++--------------------
 14 files changed, 293 insertions(+), 306 deletions(-)
 create mode 100644 sql/upgrades/155.sql

Changes applied before test

commit 7d7cb1dc01a2c92516b6ae1aad73bce07ed57a49
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 17:11:36 2020 +0200

    test_storage: Add missing tests on origin_visit_get method

commit e5e80efeedecd90794e18abbf6815edb6b991a3e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 10:57:52 2020 +0200

    storage*: Drop obsolete fields from origin_visit
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/352/ for more details.

Harbormaster completed remote builds in B13102: Diff 11888.Jun 25 2020, 11:58 AM

Rebase on latest master

Build has FAILED

Patch application report for D3349 (id=11897)

Rebasing onto 89e9dae604...

Current branch diff-target is up to date.

Changes applied before test

commit b991e697074a5aaf57f2b213baef5b42637e2893
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 17:11:36 2020 +0200

    test_storage: Add missing tests on origin_visit_get method

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/355/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/355/console

Harbormaster failed remote builds in B13112: Diff 11897!Jun 25 2020, 2:57 PM

Build is green

Patch application report for D3349 (id=11897)

Rebasing onto 89e9dae604...

Current branch diff-target is up to date.

Changes applied before test

commit b991e697074a5aaf57f2b213baef5b42637e2893
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 17:11:36 2020 +0200

    test_storage: Add missing tests on origin_visit_get method

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/357/ for more details.

Harbormaster completed remote builds in B13112: Diff 11897.Jun 25 2020, 3:15 PM

Closed by commit rDSTOb991e697074a: test_storage: Add missing tests on origin_visit_get method (authored by ardumont). · Explain WhyJun 25 2020, 3:17 PM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDSTOb991e697074a: test_storage: Add missing tests on origin_visit_get method.