Paths

Table of Contentst

Differential D3342

storage*: Drop obsolete fields from origin_visit
ClosedPublic
Actions

Authored by ardumont on Jun 24 2020, 11:00 AM.

Details

Reviewers

vlorentz

Group Reviewers

Reviewers

Maniphest Tasks

T2310: Make origin visits immutable

Commits

rDSTOe5e80efeedec: storage*: Drop obsolete fields from origin_visit

Summary

This removes:

removes those columns from the backends (cass, pg)
removes the status, metadata and snapshot from OriginVisit object instantiation
creates the initial origin-visit-status status to "created" when adding a new origin-visit

Depends on journal (test data update D3344, code adaptation D3345).

This also fixes the current failed build master [1]

[1] https://jenkins.softwareheritage.org/job/DSTO/job/tests/1299/console

Related to T2310

Test Plan

tox

Diff Detail

Repository

rDSTO Storage manager

Branch

drop-origin-visit-fields

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 13076
Build 19941: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 19940: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	20,229 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_backfill::test_backfiller swh_storage_backend_config = {'cls': 'local', 'db': 'postgresql://postgres@127.0.0.1:10902/tests', 'journal_writer': {'brokers': ['127.0.0.1:58033'], 'client_id': 'kafka_writer-1', 'cls': 'kafka', 'prefix': 'tapukxlgaj-1'}, 'objstorage': {'args': {}, 'cls': 'memory'}} kafka_prefix = 'tapukxlgaj', kafka_consumer_group = 'test-consumer-tapukxlgaj' kafka_server = '127.0.0.1:58033'
	10,019 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_replay::test_storage_play_with_collision replayer_storage_and_client = (<swh.storage.in_memory.InMemoryStorage object at 0x7f8aa6b3e2b0>, <swh.journal.client.JournalClient object at 0x7f8aa6b3e4a8>) caplog = <_pytest.logging.LogCaptureFixture object at 0x7f8aa6c3ce48>
	10,015 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_replay::test_storage_replayer replayer_storage_and_client = (<swh.storage.in_memory.InMemoryStorage object at 0x7f8c7d2bb0b8>, <swh.journal.client.JournalClient object at 0x7f8aa6be91d0>) caplog = <_pytest.logging.LogCaptureFixture object at 0x7f8aa6c53f28>
	4 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_content
	2 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.fixer::swh.storage.fixer._fix_origin
		View Full Test Results (3 Failed · 745 Passed · 17 Skipped)

Event Timeline

ardumont created this revision.Jun 24 2020, 11:00 AM

ardumont planned changes to this revision.Jun 24 2020, 11:00 AM

ardumont edited the summary of this revision. (Show Details)

ardumont edited the summary of this revision. (Show Details)Jun 24 2020, 11:03 AM

Build was aborted

Patch application report for D3342 (id=11853)

Rebasing onto 621fc8d377...

Current branch diff-target is up to date.

Changes applied before test

commit 4787c8c332254547ef541cfea22589143bb94d5d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 10:57:52 2020 +0200

    storage*: Drop obsolete fields from origin_visit
    
    Related to T2310

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/341/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/341/console

Harbormaster failed remote builds in B13068: Diff 11853!Jun 24 2020, 11:21 AM

ardumont mentioned this in D3344: journal_data: Make origin-visit optional fields to None.Jun 24 2020, 1:40 PM

ardumont edited the summary of this revision. (Show Details)Jun 24 2020, 2:36 PM

Update backfiller case

Depends on D3344 for the tests (journal test data) to pass though

Build has FAILED

Patch application report for D3342 (id=11861)

Rebasing onto 621fc8d377...

Current branch diff-target is up to date.

Changes applied before test

commit 9a5cc1d0f95707a00217fb60f469fe6a91620cb1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 10:57:52 2020 +0200

    storage*: Drop obsolete fields from origin_visit
    
    Related to T2310

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/342/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/342/console

Harbormaster failed remote builds in B13076: Diff 11861!Jun 24 2020, 2:43 PM

ardumont mentioned this in D3346: tests: Relax checks on journal writes regarding origin-visit*/snapshot.Jun 24 2020, 3:10 PM

Build has FAILED

Patch application report for D3342 (id=11861)

Rebasing onto 621fc8d377...

Current branch diff-target is up to date.

Changes applied before test

commit 9a5cc1d0f95707a00217fb60f469fe6a91620cb1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 10:57:52 2020 +0200

    storage*: Drop obsolete fields from origin_visit
    
    Related to T2310

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/347/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/347/console

ardumont added a comment.Jun 25 2020, 9:53 AM

This comment was removed by ardumont.

ardumont planned changes to this revision.Jun 25 2020, 9:53 AM

ardumont removed comment

This comment was not meant for that diff

planned changes

was not meant for that diff either... but it needs some rebase anyway and some
commented dead code to remove. So taking the opportunity here ;)

ardumont added inline comments.Jun 25 2020, 10:28 AM

swh/storage/tests/test_storage.py
2247	at it was initially the same date as the origin-visit-status from the origin-add call. This one got dropped silently `on conflict (origin, visit, date) ignore` style.

Drop unnecessary commented code

Build is green

Patch application report for D3342 (id=11881)

Rebasing onto 621fc8d377...

Current branch diff-target is up to date.

Changes applied before test

commit e5e80efeedecd90794e18abbf6815edb6b991a3e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jun 24 10:57:52 2020 +0200

    storage*: Drop obsolete fields from origin_visit
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/348/ for more details.

Harbormaster completed remote builds in B13095: Diff 11881.Jun 25 2020, 10:42 AM

ardumont edited the summary of this revision. (Show Details)Jun 25 2020, 11:03 AM

ardumont added a child revision: D3351: replayer: Fix isoformat datetime string for origin-visit.Jun 25 2020, 11:22 AM

ardumont added a child revision: D3346: tests: Relax checks on journal writes regarding origin-visit*/snapshot.Jun 25 2020, 11:35 AM

ardumont mentioned this in D3349: test_storage: Add missing tests on origin_visit_get method.Jun 25 2020, 11:50 AM

vlorentz accepted this revision.Jun 25 2020, 12:58 PM

vlorentz added inline comments.