Page MenuHomeSoftware Heritage

tests: Relax checks on journal writes regarding origin-visit*/snapshot
ClosedPublic

Authored by ardumont on Jun 24 2020, 3:10 PM.

Details

Summary

According to a recent irc discussion about the too many insertions scenario.
The gist of it was, "unnecessary to check the multiple journal writes".

So this aligns the checks with other tests to ensure we find the expected
objects in the journal in a less exhaustive way (that means no strict equality).

This diff is about aligning according the discussion and avoiding conflating
D3342's work with this.

Depends on D3342

Test Plan

tox

Diff Detail

Repository
rDSTO Storage manager
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 13079
Build 19946: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 19945: arc lint + arc unit

Unit TestsFailed

TimeTest
19,218 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_backfill::test_backfiller
swh_storage_backend_config = {'cls': 'local', 'db': 'postgresql://postgres@127.0.0.1:13153/tests', 'journal_writer': {'brokers': ['127.0.0.1:50499'], 'client_id': 'kafka_writer-1', 'cls': 'kafka', 'prefix': 'ztdpgwfqas-1'}, 'objstorage': {'args': {}, 'cls': 'memory'}} kafka_prefix = 'ztdpgwfqas', kafka_consumer_group = 'test-consumer-ztdpgwfqas' kafka_server = '127.0.0.1:50499'
10,020 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_replay::test_storage_play_with_collision
replayer_storage_and_client = (<swh.storage.in_memory.InMemoryStorage object at 0x7fcbadbf5908>, <swh.journal.client.JournalClient object at 0x7fcbadbf57b8>) caplog = <_pytest.logging.LogCaptureFixture object at 0x7fcbad393e48>
10,017 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_replay::test_storage_replayer
replayer_storage_and_client = (<swh.storage.in_memory.InMemoryStorage object at 0x7fcba8e366a0>, <swh.journal.client.JournalClient object at 0x7fcba8e2a208>) caplog = <_pytest.logging.LogCaptureFixture object at 0x7fcba8e31470>
13 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_content
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin
View Full Test Results (3 Failed · 745 Passed · 17 Skipped)

Event Timeline

Build has FAILED

Patch application report for D3346 (id=11864)

Rebasing onto 621fc8d377...

Current branch diff-target is up to date.
Changes applied before test
commit 1683a12360653b9c58203f5eb9862ab7bfc7b54e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 15:06:00 2020 +0200

    Relax checks on journal writes regarding origin-visit*

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/343/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/343/console

It's a bit sad we cannot generically use sets for this (because some model objects are still unhashable), so fine for me.

This revision is now accepted and ready to land.Jun 24 2020, 5:07 PM

It's a bit sad we cannot generically use sets for this (because some model objects are still unhashable), so fine for me.

yeah, we'll eventually be able to rework this "soon" (hopefully).

Rebase on D3342 (so builds fine)

Depends on D3342

Build is green

Patch application report for D3346 (id=11885)

Could not rebase; Attempt merge onto 621fc8d377...

Updating 621fc8d3..fd240af6
Fast-forward
 requirements-swh.txt               |   2 +-
 sql/upgrades/155.sql               |  34 +++
 swh/storage/backfill.py            |  10 +-
 swh/storage/cassandra/cql.py       |   3 -
 swh/storage/cassandra/schema.py    |   3 -
 swh/storage/cassandra/storage.py   |  15 +-
 swh/storage/db.py                  |  27 ++-
 swh/storage/in_memory.py           |  18 +-
 swh/storage/sql/30-swh-schema.sql  |  11 +-
 swh/storage/sql/40-swh-func.sql    |   4 +-
 swh/storage/sql/60-swh-indexes.sql |   2 +-
 swh/storage/storage.py             |  10 +-
 swh/storage/tests/test_backfill.py |   5 +-
 swh/storage/tests/test_storage.py  | 432 ++++++++++++++++---------------------
 14 files changed, 267 insertions(+), 309 deletions(-)
 create mode 100644 sql/upgrades/155.sql
Changes applied before test
commit fd240af6234e205b4593d4a3691fd9c01cd71ac9
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 15:06:00 2020 +0200

    Relax checks on journal writes regarding origin-visit*

commit e5e80efeedecd90794e18abbf6815edb6b991a3e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 10:57:52 2020 +0200

    storage*: Drop obsolete fields from origin_visit
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/351/ for more details.

Build is green

Patch application report for D3346 (id=11894)

Rebasing onto c6e6f33916...

Current branch diff-target is up to date.
Changes applied before test
commit 12d729b9cf3aaeb7765ca513282469f762faa243
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 15:06:00 2020 +0200

    Relax checks on journal writes regarding origin-visit*

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/353/ for more details.