tests: Relax checks on journal writes regarding origin-visit*/snapshot
ClosedPublic
Actions

Authored by ardumont on Jun 24 2020, 3:10 PM.

Details

Reviewers

olasd
douardda

Group Reviewers

Reviewers

Commits

rDSTO12d729b9cf3a: Relax checks on journal writes regarding origin-visit*

Summary

According to a recent irc discussion about the too many insertions scenario.
The gist of it was, "unnecessary to check the multiple journal writes".

So this aligns the checks with other tests to ensure we find the expected
objects in the journal in a less exhaustive way (that means no strict equality).

This diff is about aligning according the discussion and avoiding conflating
D3342's work with this.

Depends on D3342

Test Plan

tox

Diff Detail

Repository

rDSTO Storage manager

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 13079
Build 19946: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 19945: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	19,218 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_backfill::test_backfiller swh_storage_backend_config = {'cls': 'local', 'db': 'postgresql://postgres@127.0.0.1:13153/tests', 'journal_writer': {'brokers': ['127.0.0.1:50499'], 'client_id': 'kafka_writer-1', 'cls': 'kafka', 'prefix': 'ztdpgwfqas-1'}, 'objstorage': {'args': {}, 'cls': 'memory'}} kafka_prefix = 'ztdpgwfqas', kafka_consumer_group = 'test-consumer-ztdpgwfqas' kafka_server = '127.0.0.1:50499'
	10,020 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_replay::test_storage_play_with_collision replayer_storage_and_client = (<swh.storage.in_memory.InMemoryStorage object at 0x7fcbadbf5908>, <swh.journal.client.JournalClient object at 0x7fcbadbf57b8>) caplog = <_pytest.logging.LogCaptureFixture object at 0x7fcbad393e48>
	10,017 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_replay::test_storage_replayer replayer_storage_and_client = (<swh.storage.in_memory.InMemoryStorage object at 0x7fcba8e366a0>, <swh.journal.client.JournalClient object at 0x7fcba8e2a208>) caplog = <_pytest.logging.LogCaptureFixture object at 0x7fcba8e31470>
	13 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_content
	4 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin
		View Full Test Results (3 Failed · 745 Passed · 17 Skipped)

Event Timeline

ardumont created this revision.Jun 24 2020, 3:10 PM

Build has FAILED

Patch application report for D3346 (id=11864)

Rebasing onto 621fc8d377...

Current branch diff-target is up to date.

Changes applied before test

commit 1683a12360653b9c58203f5eb9862ab7bfc7b54e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 15:06:00 2020 +0200

    Relax checks on journal writes regarding origin-visit*

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/343/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/343/console

Harbormaster failed remote builds in B13079: Diff 11864!Jun 24 2020, 3:16 PM

ardumont edited the summary of this revision. (Show Details)Jun 24 2020, 4:41 PM

It's a bit sad we cannot generically use sets for this (because some model objects are still unhashable), so fine for me.

This revision is now accepted and ready to land.Jun 24 2020, 5:07 PM

It's a bit sad we cannot generically use sets for this (because some model objects are still unhashable), so fine for me.

yeah, we'll eventually be able to rework this "soon" (hopefully).

Rebase on D3342 (so builds fine)

Depends on D3342

ardumont edited the summary of this revision. (Show Details)Jun 25 2020, 11:35 AM

ardumont added a parent revision: D3342: storage*: Drop obsolete fields from origin_visit.

Build is green

Patch application report for D3346 (id=11885)

Could not rebase; Attempt merge onto 621fc8d377...

Updating 621fc8d3..fd240af6
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/155.sql               |  34 +++
 swh/storage/backfill.py            |  10 +-
 swh/storage/cassandra/cql.py       |   3 -
 swh/storage/cassandra/schema.py    |   3 -
 swh/storage/cassandra/storage.py   |  15 +-
 swh/storage/db.py                  |  27 ++-
 swh/storage/in_memory.py           |  18 +-
 swh/storage/sql/30-swh-schema.sql  |  11 +-
 swh/storage/sql/40-swh-func.sql    |   4 +-
 swh/storage/sql/60-swh-indexes.sql |   2 +-
 swh/storage/storage.py             |  10 +-
 swh/storage/tests/test_backfill.py |   5 +-
 swh/storage/tests/test_storage.py  | 432 ++++++++++++++++---------------------
 14 files changed, 267 insertions(+), 309 deletions(-)
 create mode 100644 sql/upgrades/155.sql

Changes applied before test

commit fd240af6234e205b4593d4a3691fd9c01cd71ac9
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 15:06:00 2020 +0200

    Relax checks on journal writes regarding origin-visit*

commit e5e80efeedecd90794e18abbf6815edb6b991a3e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 10:57:52 2020 +0200

    storage*: Drop obsolete fields from origin_visit
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/351/ for more details.

Harbormaster completed remote builds in B13099: Diff 11885.Jun 25 2020, 11:44 AM

Rebase on latest master

Build is green

Patch application report for D3346 (id=11894)

Rebasing onto c6e6f33916...

Current branch diff-target is up to date.

Changes applied before test

commit 12d729b9cf3aaeb7765ca513282469f762faa243
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 15:06:00 2020 +0200

    Relax checks on journal writes regarding origin-visit*

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/353/ for more details.

Harbormaster completed remote builds in B13108: Diff 11894.Jun 25 2020, 2:42 PM

Closed by commit rDSTO12d729b9cf3a: Relax checks on journal writes regarding origin-visit* (authored by ardumont). · Explain WhyJun 25 2020, 2:44 PM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDSTO12d729b9cf3a: Relax checks on journal writes regarding origin-visit*.